A dataset construction method for encrypted network traffic classification

Abstract

The application provides a dataset construction method for encrypted network traffic classification, comprising: collecting encrypted traffic data from the network according to a plurality of categories to be identified for a traffic classification task, obtaining a traffic data file, including a plurality of PCAP files under each category; determining a plurality of encrypted session traffics based on the traffic data file, including one or more encrypted session traffics contained in each PCAP file, and each session traffic including a plurality of data packets generated when the communication parties interact; extracting a plurality of modal features of each session traffic according to the data packets of each session traffic, including metadata features, encrypted payload byte features and burst transmission features; and constructing a dataset including a plurality of samples according to the plurality of modal features of the plurality of encrypted session traffics and the category corresponding to the PCAP file to which each encrypted session traffic belongs, each sample including input data constructed according to the plurality of modal features of each session traffic and a label indicating the category to which the session traffic belongs.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of network security, in particular to the field of encrypted network traffic identification, and more particularly to a method for constructing a dataset for encrypted network traffic classification. BACKGROUND

[0002] With the rapid development of network technology, network applications are emerging in an endless stream, and network security has become the focus of attention. In the field of network security, network traffic identification is crucial, especially in distinguishing various forms of encrypted traffic, including protocols, applications, services and malicious flows. However, with the widespread demand for protecting data transmission and protecting user privacy, the application scope of encryption technology has also expanded, and encrypted traffic in the network has also soared. But although encrypted traffic protects user privacy, it also provides an opportunity for those who design malicious traffic. Therefore, to cope with the challenge of traffic encryption, it is necessary to design a more robust method for classifying encrypted network traffic.

[0003] There are many methods for network traffic classification, and methods based on port numbers and methods based on deep packet inspection (Deep Packet Inspection, abbreviated as DPI) belong to rule-based methods. From the perspective of availability, after traffic encryption, due to port concealment and difficulty in feature extraction, some traditional traffic identification methods based on rules are no longer applicable, such as reference [1] and reference [2]; from the perspective of security, the principles of these traditional methods are very clear, which also provides convenience for attackers to design more "realistic" malicious attack traffic. Some methods attempt to use the handshake information of the Transport Layer Security (TLS) 1.2 version of the TLS to identify traffic, but with the widespread use of the TLS 1.3 version of the encryption protocol, some handshake information is not visible.

[0004] In recent years, more and more work has begun to explore what features can be used for network traffic classification after traffic encryption. Because deep learning can directly extract and identify features from raw encrypted data, thus providing a high-precision end-to-end solution, more and more encrypted traffic identification methods based on deep learning have been proposed, such as references [3], [4], [5] and [6]. In addition, there are also some deep learning methods as follows:

[0005] As reference [7] proposed by Lotfollahi M et al., which is based on the Deep Packet method implemented by deep learning, can process network traffic classification into traffic characterization of major categories (such as FTP and P2P), and application identification of terminal user application (such as BitTorrent and Skype) identification, not only can identify encrypted traffic, but also can distinguish VPN network traffic and non-VPN network traffic, but because its model design is relatively simple, Deep Packet, although using deep learning method to achieve the goal of encrypted traffic identification, but this method performs very poorly in Tor traffic classification, so it still needs to be greatly improved.

[0006] As reference [8] proposed by C. Liu et al., it is a flow sequence network FS-Net for encrypted traffic identification, which uses packet length sequence as the input of the network, and automatically realizes feature extraction and output. This method promotes feature learning through reconstruction mechanism, and generates features that can contain more discriminative information by keeping the reconstructed sequence and the original flow sequence as similar as possible, so as to improve the identification ability. But FS-Net only uses one kind of traffic feature, packet length sequence, which is easy to be affected by packet loss, out-of-order and other traffic camouflage techniques, and has poor robustness.

[0007] As reference [9] proposed by T. Shapira and Y. Shavitt, it is FlowPic method, which uses the characteristics related to time and packet size in the flow to convert basic traffic data into intuitive FlowPic pictures, and then uses known image classification deep learning technology CNN to identify traffic categories (such as browsing, chatting, video traffic categories, etc.). This method uses single feature, and is still affected by packet loss, out-of-order and other traffic camouflage techniques, and has poor anti-interference ability.

[0008] Considering that packet length sequence, packet arrival time interval sequence and other metadata features may be affected by traffic camouflage, some works find that the plaintext information of network flow is not completely randomized, but still has some differences, such as reference

[10] , the plaintext information of network flow contains some semantic information, which can also be used for network traffic classification. Therefore, some methods using natural language processing model to learn the semantic information of encrypted traffic payload to identify encrypted traffic have also appeared.

[0009] Meanwhile, in deep learning, the quality and quantity of input samples are also crucial to the model effect. However, the models built by supervised learning methods such as CNN and RNN have a high degree of dependence on labeled data. However, labeled data not only has a small amount of data, but also has complex manual annotation, and may also have annotation errors and other problems, so that these supervised learning methods can achieve good results on the training set, but the effect is not good when deployed in real network scenarios. Therefore, combined with the characteristics of the encrypted payload information mentioned above, such as the reference

[11] proposed by Lin X, which is an encrypted traffic recognition method ET-BERT based on the natural language processing model BERT, uses natural language processing methods to process and learn encrypted payload information, and uses an unsupervised pre-training method. However, encrypted traffic is very different from natural language and does not contain explicit semantic information. ET-BERT only uses encrypted payload byte information, and the semantic features learned are weak, and the encrypted traffic classification effect is not ideal.

[0010] Therefore, when classifying encrypted network traffic, the existing method is difficult to extract features based on rules, which has led to the fact that it is no longer applicable to extract feature data for encrypted traffic, and the encrypted information learned based on deep learning has the problems of large amount of data required, limited and single feature information that can be learned, and thus the encrypted traffic classification model has the problems of poor classification effect and low robustness.

[0011] It should be noted that: the background art is only used to introduce the related information of the present application, so as to help understand the technical solutions of the present application, but does not mean that the related information must be prior art. The related information is submitted and disclosed together with the present application scheme, and in the absence of evidence that the related information has been disclosed before the filing date of the present application, the related information should not be regarded as prior art.

[0012] The references are as follows:

[0013] [1] V. F. Taylor, R. Spolaor, M. Conti, and I. Martinovic, ”Robust smartphone app identification via encrypted network traffic analysis,” IEEE Transactions on Information Forensics and Security, vol. 13, no. 1, Jan. 2018, pp. 63-78.

[0014] [2] Y. Chen, T. Zang, Y. Zhang, Y. Zhou, and Y. Wang, “Rethinking encrypted traffic classification: A multi-attribute associated fingerprint approach,” in 27th IEEE International Conference on Network Protocols, ICNP 2019, Chicago, IL, USA, October 8-10, 2019. IEEE, 2019, pp. 1-11.

[22] J. Che

[0015] [3] T. Ongun, T. Sakharaov, S. Boboila, A. Oprea, and T. Eliassi-Rad, “On designing machine learning models for malicious network traffic classification,” CoRR, vol. abs / 1907.04846, 2019.

[0016] [4] Rezaei S, Liu X. Deep Learning for Encrypted Traffic Classification: An Overview[J]. IEEE Communications Magazine, 2019, 57(5):76-81.

[0017] [5] Lan J, Liu X, Li B, et al. DarknetSec: A novel self-attentive deep learning method for darknet traffic classification and application identification[J]. Computers & Security, 2022, 116(2):102663.

[0018] [6] J. Guo et al., "Global-Aware Prototypical Network for Few-Shot Encrypted Traffic Classification," 2022 IFIP Networking Conference (IFIP Networking), Catania, Italy, 2022, pp. 1-9, doi: 10.23919 / IFIPNetworking55013.2022.9829771.

[0019] [7] Liu C, He L, Xiong G, et al. Fs-net: A flow sequence network for encrypted traffic classification [C] / / IEEE INFOCOM 2019-IEEE Conference On Computer Communications. IEEE, 2019: 1171-1179.

[0020] [8] Lotfollahi M, Zade R S H, Siavoshani M J, et al. Deep Packet: A Novel Approach For Encrypted Traffic Classification Using Deep Learning [J]. Soft Computing, 2017.

[0021] [9] T. Shapira and Y. Shavitt, "FlowPic: A Generic Representation for Encrypted Traffic Classification and Applications Identification," in IEEE Transactions on Network and Service Management, vol. 18, no. 2, pp. 1218-1232, June 2021, doi: 10.1109 / TNSM.2021.3071441.

[0022]

[10] Xinjie Lin, Gang Xiong, Gaopeng Gou, Zhen Li, Junzheng Shi, JingYu: ETBERT: A Contextualized Datagram Representation with Pretraining Transformers for Encrypted Traffic Classification. WWW 2022.

[0023]

[11] Lin X, Xiong G, Gou G, et al. Et-bert: A contextualized datagram representation with pre-training transformers for encrypted traffic classification[C] / / Proceedings of the ACM Web Conference 2022. 2022: 633-642. SUMMARY

[0024] Therefore, the purpose of the present application is to overcome the defects of the prior art, and provide a dataset construction method for encrypted network traffic classification.

[0025] The purpose of the present application is achieved by the following technical solutions:

[0026] According to a first aspect of the present application, a dataset construction method for encrypted network traffic classification is provided, comprising: S1, collecting encrypted traffic data from the network according to a plurality of categories identified as required by a traffic classification task, to obtain a traffic data file, which includes a plurality of original PCAP files corresponding to each category in the plurality of categories for recording encrypted traffic data; S2, determining a plurality of encrypted session traffics based on the traffic data file, including determining one or more encrypted session traffics contained in each original PCAP file according to each original PCAP file, each encrypted session traffic including a plurality of data packets generated in the interaction process of the communication parties; S3, extracting a plurality of modal features of each encrypted session traffic according to the data packets of the encrypted session traffic, including metadata features, encrypted payload byte features and burst transmission features; S4, constructing a dataset including a plurality of samples according to the plurality of modal features of the plurality of encrypted session traffics and the category corresponding to the original PCAP file to which each encrypted session traffic belongs, each sample including input data constructed according to the plurality of modal features of each encrypted session traffic and a label indicating the category to which the encrypted session traffic belongs.

[0027] In some embodiments of the present application, in the S2, determining one or more encrypted session traffics contained in each original PCAP file according to the original PCAP file comprises: extracting original information from the original PCAP file, the original information comprising a plurality of data packets and packet header fields of each data packet; determining one or more encrypted session traffics in the form of sessions according to the packet header fields of each data packet, and dividing the plurality of data packets into the encrypted session traffics to which the plurality of data packets belong, wherein the encrypted session traffic in the form of a session comprises bidirectional traffic formed when two parties communicate bidirectionally.

[0028] In some embodiments of the present application, the packet header field comprises five-tuple information composed of a source IP address, a destination IP address, a source port number, a destination port number and a transport layer protocol, and the manner of dividing the plurality of data packets into the encrypted session traffics to which the plurality of data packets belong comprises: determining IP addresses and port numbers interacted by each pair of communication parties according to the five-tuple information of each data packet; and dividing data packets related to the IP addresses and port numbers interacted by a pair of communication parties into the corresponding encrypted session traffic according to the five-tuple information of each data packet.

[0029] In some embodiments of the present application, in the S3, the metadata features comprise a plurality of timing data, and the manner of extracting the plurality of timing data comprises: obtaining a preset number of data packets in each encrypted session traffic in chronological order; extracting metadata of each data packet in the obtained preset number of data packets, the metadata comprising a byte length of the data packet and an arrival time interval, the arrival time interval being determined according to a difference between a time at which the data packet arrives and a time at which a time-neighboring data packet before the data packet arrives; and obtaining the plurality of timing data according to the metadata of all the obtained data packets, the plurality of timing data comprising splicing the byte length of each data packet and the arrival time interval to obtain a timing data.

[0030] In some embodiments of the present application, in the S3, the encrypted payload byte feature in the multi-modal feature is extracted in the following manner: a preset number of bytes in each encrypted session traffic are extracted, and all the extracted bytes do not include bytes of a packet header and bytes of a packet tail; the preset number of bytes are constructed into a payload byte image of a preset size, and the byte image is taken as the encrypted payload byte feature, wherein a value of each of the preset number of bytes is taken as a value of each pixel in the byte image.

[0031] In some embodiments of the present application, in the S3, the extraction manner of the burst transmission feature in the multi-modal feature comprises: acquiring, in chronological order, a preset number of data packets in each encrypted session traffic, and counting a plurality of burst transmission processes existing in all the acquired data packets, each burst transmission process representing a continuous transmission process of a plurality of data packets performed in the same communication direction; counting the number of bytes transmitted by each of the plurality of burst transmission processes, the number of bytes transmitted by each burst transmission process being the sum of the number of bytes of all data packets transmitted by the burst transmission process; counting the time interval corresponding to each of the plurality of burst transmission processes, the time interval corresponding to each burst transmission process being the difference between the end time of the burst transmission process and the end time of the previous adjacent burst transmission process; generating a two-dimensional joint distribution graph according to the number of bytes transmitted by each of the plurality of burst transmission processes and the corresponding time interval, the two-dimensional joint distribution graph comprising a plurality of value pairs, each value pair recording the number of bytes transmitted by a burst transmission process and the corresponding time interval; mapping the two-dimensional joint distribution graph into a burst distribution image of a preset size, and taking the distribution graph as the burst transmission feature.

[0032] In some embodiments of the present application, in the S4, the multi-modal feature of one encrypted session traffic is used to construct the input data of a sample, and the construction manner comprises: taking the metadata feature of one encrypted session traffic as a time feature, splicing the encrypted payload byte feature and the burst transmission feature of the encrypted session traffic to obtain the spatial feature of the traffic; constructing the input data of the sample, which comprises the time feature and the spatial feature of the encrypted session traffic.

[0033] In some embodiments of the present application, in the S1, the traffic classification task is an application traffic identification task, a service traffic identification task or a malicious traffic identification task, wherein the plurality of categories required to be identified by the application traffic identification task comprises a plurality of application corresponding traffic categories; the plurality of categories required to be identified by the service traffic identification task comprises a plurality of service corresponding traffic categories; and the plurality of categories required to be identified by the malicious traffic identification task comprises a malicious traffic category and a normal traffic category.

[0034] According to a second aspect of the present application, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is configured to store executable instructions; and the one or more processors are configured to implement the steps of the method of the first aspect of the present application by executing the executable instructions.

[0035] Compared with the prior art, the present application has the following advantages:

[0036] The method of the application collects multiple original PCAP files for each traffic category to ensure the integrity and diversity of samples in the dataset; secondly, each encrypted session traffic includes interactive information of both parties, expressing more information, which is more conducive to subsequent feature extraction and traffic classification; then, the multi-modal features of each encrypted session traffic are extracted, including metadata features, encrypted payload byte features and burst transmission features, so that the dataset constructed by the application includes multi-dimensional and rich traffic feature information; finally, each sample in the dataset includes input data constructed according to the multi-modal features of each encrypted session traffic, that is, it includes multi-dimensional encrypted traffic features, which is more conducive to the model learning more traffic features and semantic information when applied to the training of a deep learning model, so as to improve the robustness of the model. BRIEF DESCRIPTION OF DRAWINGS

[0037] The embodiments of the application will be further described below with reference to the drawings, in which:

[0038] Figure 1 The flowchart of the dataset construction method according to the embodiments of the application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below with reference to the drawings through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application.

[0040] As mentioned in the background section, when classifying encrypted network traffic, the rule-based method is very difficult to extract features, which has led to its inapplicability to extract feature data for encrypted traffic, and the encrypted information learned based on deep learning has the problems of large amount of required data, limited and single learnable feature information, which further leads to the problems of poor classification effect and low robustness of the encrypted traffic classification model.

[0041] To solve the above problems, the inventors have found from multiple angles that: 1) the rule-based method extracts the features of encrypted traffic, i.e. metadata features, which, although affected by traffic camouflage, cannot be completely denied its role; 2) the semantic information of encrypted payload bytes, although relatively weak, cannot be denied its role; 3) burst transmission reflects the behavior information of traffic. Since some applications will cache the data packets to be sent in advance in order to reduce the burden on the CPU, and then send them together when the cache reaches a certain amount, the application layer always tends to send or receive as much data as possible at a time, thus forming burst transmission. Different types of traffic have different burst transmissions, so the burst feature can also be an important feature for identifying encrypted network traffic.

[0042] Based on the above research and analysis, according to an embodiment of the present application, a data set construction method for encrypted network traffic classification is provided. First, for each traffic category to be identified for the traffic classification task, the method of the present application collects a plurality of original PCAP files for recording encrypted traffic, to ensure the integrity and diversity of the samples in the data set; second, the encrypted session traffic in the form of a session is determined in each original PCAP file, each encrypted session traffic includes the interaction information of the two parties in communication, and the information expressed is more conducive to subsequent feature extraction and traffic classification; then, the multi-modal features of each encrypted session traffic are extracted, including metadata features, encrypted payload byte features and burst transmission features, so that the data set constructed by the present application includes multi-dimensional and rich traffic feature information; finally, each sample in the data set includes input data constructed according to the multi-modal features of each encrypted session traffic and a label indicating the category to which the encrypted session traffic belongs. The encrypted traffic features in the constructed data set are multi-dimensional, and when applied to the training of a deep learning model, compared with the training and learning method using only metadata features or encrypted payload byte features, the data set constructed by the present application is more conducive to the model learning more traffic features and semantic information, to improve the robustness of the model.

[0043] In general, according to an embodiment of the present application, the construction of the data set of the present application can refer to Figure 1 , which is a data set construction method flowchart. It includes steps S1, S2, S3 and S4. After the construction of the data set is completed, the data set can be applied to train a model for encrypted network traffic classification, to obtain a model that can perform encrypted network traffic classification. In order to better understand the present application, the construction method of the data set of the present application and the application of the data set are described in detail below in connection with specific embodiments.

[0044] I. Data set construction method

[0045] In step S1, according to the multiple categories to be identified for the traffic classification task, encrypted traffic data is collected from the network to obtain traffic data files, which include a plurality of original PCAP files for recording encrypted traffic data under each category in the multiple categories.

[0046] According to an embodiment of the present application, the traffic classification task is an application traffic identification task, a service traffic identification task, or a malicious traffic identification task, and the present application is not limited thereto. The multiple categories to be identified in the application traffic identification task include traffic categories corresponding to multiple applications, the multiple categories to be identified in the service traffic identification task include traffic categories corresponding to multiple services, and the multiple categories to be identified in the malicious traffic identification task include malicious traffic categories and normal traffic categories. Illustratively, the traffic categories corresponding to multiple applications can include WeChat application traffic categories, Taobao application traffic categories, Douyin application traffic categories, Meituan application traffic categories, and the like; and the traffic categories corresponding to multiple services can include live streaming traffic categories, on-demand traffic categories, voice traffic categories, download traffic categories, game traffic categories, music traffic categories, and the like.

[0047] According to an embodiment of the present application, taking the application traffic identification task as an example, a tool such as Wireshark or Tshark can be used to capture traffic data packets under each application in the network and save them as original PCAP files. The original PCAP file of the application traffic records the application to which the original PCAP file belongs when the tool is used to capture packets. In order to distinguish the application traffic category to which the encrypted network traffic recorded by the file belongs.

[0048] In step S2, a plurality of encrypted session traffics are determined based on the traffic data files, including determining one or more encrypted session traffics contained in each original PCAP file, and each encrypted session traffic includes a plurality of data packets generated in the interaction process of the communication parties.

[0049] According to an embodiment of the present application, in step S2, the manner of determining one or more encrypted session traffics contained in each original PCAP file includes steps S21 and S22:

[0050] Step S21: Extracting original information from the original PCAP file, which includes a plurality of data packets and packet header fields of each data packet.

[0051] According to an embodiment of the present application, the contents of each original PCAP file are read one by one to extract the original information of each original PCAP file. The original information includes all data packets contained in the PCAP file and each data packet information, and each data packet information includes a source IP address, a destination IP address, a source port number, a destination port number, a transmission layer protocol, a byte length of the data packet, and a time when the data packet arrives. The packet header field includes five-tuple information composed of the source IP address, the destination IP address, the source port number, the destination port number, and the transmission layer protocol.

[0052] Step S22: determining one or more encrypted session traffics in the form of sessions according to the packet header fields of the data packets, and dividing the data packets into the encrypted session traffics to which the data packets belong, wherein the encrypted session traffic in the form of a session includes bidirectional traffics formed when the two parties communicate with each other.

[0053] According to one embodiment of the present application, the encrypted session traffic includes bidirectional traffics, all data packets of each of the bidirectional traffics have the same five-tuple information, and the encrypted session traffic includes all data packets of the bidirectional traffics. Illustratively, when the two parties of communication are Party A and Party B respectively, the bidirectional traffics include one traffic corresponding to data transmission from Party A to Party B and one traffic corresponding to data transmission from Party B to Party A. The technical solution of this embodiment can at least achieve the following beneficial technical effects: since the session traffic contains bidirectional traffics, which include interactive information of the two parties of communication, more information can be obtained, and the traffic classification is more beneficial.

[0054] According to one embodiment of the present application, the manner of dividing the data packets into the encrypted session traffics to which the data packets belong includes: determining pairs of IP addresses and port numbers interacted by each pair of the two parties of communication according to the five-tuple information of each data packet; and dividing data packets related to the pairs of IP addresses and port numbers interacted by a pair of the two parties of communication into the corresponding encrypted session traffic according to the five-tuple information of each data packet. That is, the five-tuple information of all data packets in the encrypted session traffic includes the pairs of IP addresses and port numbers interacted by a pair of the two parties of communication. And the source IP address and the destination IP address in the five-tuple information of a data packet in the encrypted session traffic are still the data packets in the encrypted session traffic after being interchanged, and the source port number and the destination port number are also interchanged.

[0055] In step S3, multi-modal features of each encrypted session traffic are extracted from the data packets of the encrypted session traffic, which include metadata features, encrypted payload byte features and burst transmission features.

[0056] According to one embodiment of the present application, the extraction manners of the metadata features, the encrypted payload byte features and the burst transmission features are described as follows:

[0057] 1) Extraction of metadata features

[0058] According to one embodiment of the present application, the metadata feature includes a plurality of time series data, and the extraction manner of the plurality of time series data includes: obtaining, in chronological order, a preset number of data packets in each encrypted session flow; extracting metadata of each data packet in the obtained preset number of data packets, including byte length of the data packet and arrival time interval, the byte length of each data packet including byte length occupied by all data in the packet, and the arrival time interval being determined according to a difference between a time at which the data packet arrives and a time at which a time-neighboring data packet before the data packet arrives; and obtaining the plurality of time series data according to the metadata of all the data packets, including splicing the byte length and the arrival time interval of each data packet to obtain a time series data.

[0059] According to one embodiment of the present application, n data packets transmitted in chronological order in each encrypted session flow are obtained. It should be noted that when the three features of metadata feature, encrypted payload byte feature and burst transmission feature are extracted, the data packets and the total number of data packets used are the same, which can be 500 data packets transmitted in chronological order in each encrypted session flow, 1000 data packets transmitted in chronological order in each encrypted session flow, or 1200 data packets transmitted in chronological order in each encrypted session flow, and the present application does not limit the value of n.

[0060] According to one embodiment of the present application, metadata of each data packet in the obtained n data packets is extracted, including counting byte length and arrival time interval of the n data packets transmitted in chronological order in each encrypted session flow. When n = 1000, byte length of 1000 data packets transmitted in chronological order in each encrypted session flow is counted respectively to obtain a byte sequence including 1000 byte lengths, and arrival time interval of the 1000 data packets transmitted in chronological order in each encrypted session flow is counted respectively to obtain a time sequence including 1000 arrival time intervals, the byte sequence and the time sequence are spliced one by one to form a metadata feature including 1000 time series data and each time series data having a dimension of 2. The present application extracts time series data as metadata feature to realize analysis of flow type from the time angle.

[0061] 2) Extraction of encrypted payload byte feature

[0062] According to one embodiment of the present invention, a method for extracting an encrypted payload byte feature from a multimodal feature includes extracting a preset number of bytes from each encrypted session flow, excluding bytes in the packet header and the packet trailer. The extracted bytes are encrypted payload bytes, which refer to the actual transmitted data located in the middle of a packet after removing the header and trailer bytes in network communications. The extracted preset number of bytes are constructed into a payload byte image of a preset size, and the byte image is used as the encrypted payload byte feature. The values ​​of the preset number of bytes are used as the values ​​of each pixel in the byte image.

[0063] According to one embodiment of the present invention, the first 1000 packets transmitted during each encrypted session are obtained. After removing the header and tail bytes from each packet, bytes are collected from the 1000 packets without the header and tail bytes removed. For example, constructing a payload byte image with a preset size of 256×256×1 (length×width×number of channels, the same below) is used to illustrate: 65,536 bytes are collected from the 1000 packets without the header and tail bytes removed. If the bytes in the 1000 packets without the header and tail bytes are too long, the bytes of the subsequent packets are trimmed, leaving only the first 65,536 bytes. If the number of bytes is less than 65,536, all bytes in the 1000 packets without the header and tail bytes are padded with zeros until 65,536 bytes are met. Using 65,536 bytes, and reconstructing an image with a height of 256 and a width of 256, with one byte corresponding to one pixel in the image, a payload byte image with dimensions of 256 × 256 × 1 is generated. Since each byte is a value between 0 and 255, the pixels in this payload byte image are values ​​between 0 and 255. This payload byte image is used as the encrypted payload byte feature. If a payload byte image with a preset size of 128 × 128 × 1 is constructed, 16,384 bytes must be collected from the 1,000 data packets, after removing the header and tail bytes, to construct the image, as described above.

[0064] 3) Extraction of Burst Transmission Features

[0065] According to one embodiment of the present invention, a method for extracting burst transmission characteristics includes steps a1, a2, a3, a4, and a5:

[0066] Step a1: Obtain a preset number of data packets from each encrypted session in chronological order, and count the number of burst transmission processes present in all the acquired data packets. Each burst transmission process represents the continuous transmission process of multiple data packets in the same communication direction.

[0067] According to one embodiment of the present application, a burst transmission indicates a continuous transmission of multiple data packets in the same communication direction, and each encrypted session traffic is bidirectional traffic, such as the encrypted session traffic generated when A and B communicate with each other. 1000 data packets are obtained in the previous transmission of the encrypted session traffic, and the transmission from A to B and the transmission from B to A are two directions. If the first to fifth data packets in the obtained 1000 data packets are all continuously transmitted from A to B, then the process of transmitting the first to fifth data packets constitutes a burst transmission process. In this way, the number of burst transmission processes existing in the obtained 1000 data packets is counted.

[0068] Step a2: Count the number of bytes transmitted by each of the multiple burst transmission processes. The number of bytes transmitted by each burst transmission process is the sum of the number of bytes of all data packets transmitted by the burst transmission process.

[0069] According to one embodiment of the present application, if a burst transmission process includes 5 continuously transmitted data packets, then the number of bytes of the 5 data packets is summed to obtain the number of bytes transmitted by the burst transmission process. When counting, the number of bytes transmitted by each of the multiple burst transmission processes is counted in the time sequence of all burst transmission processes. For example, 1000 data packets are taken as an example. The process of continuously transmitting the first to eighth data packets in the same communication direction is the first burst transmission process, the process of continuously transmitting the ninth to fifteenth data packets in the same communication direction is the second burst transmission process, and the process of continuously transmitting the sixteenth to twenty-second data packets in the same communication direction is the third burst transmission process. For example, the number of bytes transmitted by the first, second, and third burst transmission processes is counted in sequence as 64, 128, and 32, respectively, to obtain the number of bytes transmitted by each of the multiple burst transmission processes in the 1000 data packets.

[0070] Step a3: Count the time interval corresponding to each of the multiple burst transmission processes. The time interval corresponding to each burst transmission process is the difference between the end time of the burst transmission process and the end time of the previous adjacent burst transmission process.

[0071] According to one embodiment of the present application, for example, in the first, second, and third burst transmission processes given in the above embodiment, the time interval of the first burst transmission process is the difference between the end time of the burst transmission process and 0, the time interval of the second burst transmission process is the difference between the end time of the burst transmission process and the end time of the first burst transmission process, and the time interval of the third burst transmission process is the difference between the end time of the burst transmission process and the end time of the second burst transmission process. The time interval corresponding to each of the multiple burst transmission processes is calculated in this way.

[0072] Step a4: generating a two-dimensional joint distribution graph according to the number of bytes transmitted by each burst transmission process and the corresponding time interval, which includes a plurality of value pairs, each value pair recording the number of bytes transmitted by a burst transmission process and the corresponding time interval.

[0073] According to an embodiment of the present application, the number of bytes transmitted by each burst transmission process is normalized, and the corresponding time interval of each burst transmission process is normalized, to generate a two-dimensional joint distribution graph (two-dimensional graph) of the number of bytes and the time interval of the plurality of burst transmission processes. The normalization is to map the values to the interval [0, 1], and each value pair in the two-dimensional joint distribution graph records the normalized number of bytes and the normalized time interval corresponding to a burst transmission process in two-dimensional form. For example, assuming that the horizontal axis of the graph is the x-axis and the vertical axis is the y-axis, then the x-axis represents the de-duplicated and ordered sequence of the normalized number of bytes of each burst transmission process, and the y-axis represents the de-duplicated and ordered sequence of the normalized time interval of each burst transmission process. The de-duplicated and ordered sequence means the reordered sequence after removing the repeated values, for example, the original sequence of the normalized number of bytes of each burst transmission process is [0.1, 0.5, 0.2, 0.8, 0.2], and after de-duplication and ordering, it becomes [0.1, 0.2, 0.5, 0.8]. Then any point on the two-dimensional graph corresponds to a two-dimensional value pair, for example, the i-th point corresponds to the value pair (Xi, Yi), where Xi is the normalized number of bytes of the burst transmission process corresponding to the i-th point, and Yi is the normalized time interval of the burst transmission process corresponding to the i-th point.

[0074] Step a5: mapping the two-dimensional joint distribution graph into a preset size burst distribution image, and taking the distribution graph as a burst transmission feature.

[0075] According to an embodiment of the present application, the two-dimensional joint distribution graph can be proportionally mapped into a 256 high and 256 wide. Thus, a burst distribution image is obtained, and the picture size is 256x256x1, where 1 is the number of channels. The distribution graph is taken as a burst transmission feature. In addition, the preset size of the payload byte image and the burst distribution image is generally set to be the same, which can be both 256x256x1, both 128x128x1, or both 64x64x1, and the present application does not limit this. In order to facilitate subsequent construction of a spatial feature according to the payload byte feature and the burst transmission feature.

[0076] In step S4, a dataset including a plurality of samples is constructed according to the multi-modal features of the plurality of pieces of encrypted session traffic and the categories corresponding to the original PCAP files to which each piece of encrypted session traffic belongs, each sample including input data constructed according to the multi-modal features of each piece of encrypted session traffic and a label indicating the category to which the encrypted session traffic belongs.

[0077] According to an embodiment of the present application, the input data of a sample is constructed using the multi-modal features of a piece of encrypted session traffic, and the construction includes: taking the metadata features of a piece of encrypted session traffic as time features, concatenating the encrypted payload byte features and burst transmission features of the encrypted session traffic to obtain the spatial features of the traffic; and constructing the input data of the sample, which includes the time features and the spatial features of the encrypted session traffic.

[0078] According to an embodiment of the present application, the metadata features are taken as time features, the encrypted payload byte features are taken as one channel of a picture, and the burst transmission features are extracted as another channel of the picture, to jointly generate a two-channel picture as spatial features. Illustratively, the metadata features of each piece of encrypted session traffic are taken as time features, and 1000 data packets transmitted in advance of each piece of encrypted session traffic is taken as an example, the time features being a multi-channel time sequence with a length of 1000 and a channel number of 2. The obtained 256x256x1 payload byte image and 256x256x1 burst distribution image are concatenated in the channel dimension, to obtain a picture with a height and a width of 256 and a channel number of 2 (the picture dimension is denoted as 256x256x2), which is the spatial features of the input data of the sample. Thus, the spatial feature input and the time feature input of each sample can be obtained. The technical solution of this embodiment can at least achieve the following beneficial technical effects: the input data of the sample of the present application includes multi-modal traffic feature information, which is beneficial to the model to learn traffic features and semantic information from multiple dimensions, so as to improve the traffic recognition performance of the model.

[0079] II. Application of the dataset

[0080] According to an embodiment of the present application, the dataset obtained in the above embodiments can be used to train a model for classifying encrypted network traffic, and the application implementation process of the dataset is described as follows.

[0081] Embodiment 1

[0082] According to an embodiment of the present application, a training method for an encrypted network traffic classification model is provided, including a pre-training phase and a downstream task training phase, wherein the encrypted network traffic classification model includes a deep learning model and a classifier, the deep learning model is trained through the pre-training phase to obtain a pre-trained deep learning model, and the parameters of the pre-trained deep learning model are frozen through the downstream task training phase to train the classifier. The deep learning model can be formed by stacking Transformers or constructed by other deep neural networks, and the present application is not limited thereto.

[0083] According to an embodiment of the present application, the process of training the deep learning model in the pre-training phase includes the following processes:

[0084] 1) A first training set is constructed based on the data set constructed by the method of the above embodiment, including a plurality of positive samples and negative samples, the input data of each positive sample includes time features and spatial features from the same encrypted session traffic, the input data of each negative sample includes time features and spatial features from different encrypted session traffic, and the first label of each positive sample and negative sample is used to indicate whether the time features and spatial features of its input data match. The first label of the positive sample is marked as 1 to indicate matching, and the first label of the negative sample is marked as 0 to indicate mismatching.

[0085] 2) The deep learning model is trained using the first training set to process the time features and spatial features of the input data of each sample in the first training set respectively to obtain the time encoding result and the spatial encoding result of each sample in the first training set, and the matching degree of the time features and spatial features of each sample in the first training set is obtained according to the time encoding result and the spatial encoding result of the sample, and the parameters of the deep learning model are updated to minimize the difference between the matching degree of each sample in the first training set and the first label of the sample as the optimization goal to obtain the pre-trained deep learning model.

[0086] According to one embodiment of the present application, the deep learning model comprises a first deep neural network module and a second deep neural network module, the time features of the input data of each sample in the first training set are encoded by the first deep neural network module to obtain time encoding results, and the space features of the input data of each sample in the first training set are encoded by the second deep neural network module to obtain space encoding results. The matching degree of the time features and the space features can be obtained by calculating the similarity between the time encoding results and the space encoding results of each sample in the first training set. The parameters of the deep learning model are updated by maximizing the similarity between the time encoding results and the space encoding results of the positive samples and minimizing the similarity between the time encoding results and the space encoding results of the negative samples as the optimization target to obtain the pre-trained deep learning model. The pre-training process can make the deep learning model learn the deep semantic correlation between the space features and the time features.

[0087] According to one embodiment of the present application, the process of training the classifier in the downstream task training stage comprises the following processes:

[0088] 3) Obtain the pre-trained deep learning model of the above embodiment and the second training set constructed based on the dataset constructed by the method of the above embodiment, comprising a plurality of samples, each sample comprising input data constructed according to the multi-modal features extracted from each encrypted session traffic and a label indicating the traffic type to which the encrypted session traffic belongs, and the input data comprising the time features and the space features of the same encrypted session traffic.

[0089] 4) Train the classifier using the second training set, comprising the following steps:

[0090] 4.1) Duplicate the time features of each sample of the second training set multiple times, and respectively splice different traffic category attributes [CLS] in front of each time feature, and then splice with a randomly initialized soft prompt to obtain multiple preprocessed time features. The multiple preprocessed time features of each sample are input into the first deep neural network module of the pre-trained deep learning model for encoding to obtain multiple time encoding results. Wherein, the encrypted network traffic classification model needs to recognize how many categories, and needs to duplicate the time features of each sample how many times.

[0091] 4.2) input the spatial features of each sample of the second training set into the second deep neural network module of the pre-trained deep learning model for encoding to obtain a spatial encoding result, input the spatial encoding result into the classifier to obtain a model output result, which includes probability values of the sample belonging to each traffic category. The probability values are obtained by calculating the similarity between the spatial encoding result and each of the plurality of time encoding results, and the traffic category attribute corresponding to the time encoding result with the highest similarity with the spatial encoding result is taken as the traffic category of the sample.

[0092] 4.3) freeze the parameters of the pre-trained deep learning model, and update the parameters of the classifier with the optimization objective of minimizing the difference between the model output result and the label.

[0093] The technical solution of the embodiment 1 at least achieves the following beneficial effects: the pre-training phase can adjust the same deep learning model for different downstream tasks, and only a small number of parameters need to be fine-tuned in the downstream task training phase, thereby enhancing the applicability of the deep learning model to a wide range of downstream tasks.

[0094] Embodiment 2

[0095] According to another embodiment of the present application, another method for training an encrypted network traffic classification model is provided, comprising: obtaining a data set constructed based on the method of the above embodiments, which includes a plurality of samples, each sample including input data constructed according to the multi-modal features extracted from each encrypted session traffic and a label indicating the traffic type to which the encrypted session traffic belongs; training an encrypted network traffic classification model using the data set to learn to identify the traffic type to which the encrypted session traffic belongs according to the input data, and obtaining an identification result of each encrypted session traffic, and updating the parameters of the model with the optimization objective of minimizing the difference between the identification result and the label.

[0096] It should be noted that although the above describes the steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of the steps can be performed concurrently or even in a different order, as long as the desired function can be achieved.

[0097] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions embodied therewith, which cause a processor to carry out various aspects of the present application.

[0098] A computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves.

[0099] Embodiments of the application have been described above, with the understanding that these embodiments are exemplary only, and are not restrictive, in terms of the scope of the embodiments disclosed. Many modifications and variations of the described embodiments are possible, in light of the above teachings, without departing from the scope and spirit of the described embodiments. The choice of words in this document is intended to best explain the principles of the embodiments, practical application, or technical improvement in the art, or to enable others skilled in the art to utilize the embodiments disclosed herein.

Claims

1. A method for constructing a dataset for encrypted network traffic classification, characterized in that: include: S1. Collect encrypted traffic data from the network according to multiple categories required for identification in the traffic classification task, and obtain a traffic data file, which includes multiple original PCAP files corresponding to each of the multiple categories for recording the encrypted traffic data; S2. Determining a plurality of encrypted session flows based on the flow data file includes determining, based on each original PCAP file, one or more encrypted session flows contained in the PCAP file, each encrypted session flow including a plurality of data packets generated during an interaction between two communicating parties. In S2, determining, based on each original PCAP file, the one or more encrypted session flows contained in the PCAP file includes: Extracting original information from the original PCAP file, the original information including a plurality of data packets and a header field of each data packet; Determine one or more encrypted session flows in the form of a session based on a header field of each data packet, and classify the multiple data packets into the encrypted session flows. The encrypted session flows in the form of a session include bidirectional flows generated during bidirectional communication between two communicating parties. The header field includes five-tuple information consisting of a source IP address, a destination IP address, a source port number, a destination port number, and a transport layer protocol. Methods for classifying the multiple data packets into the encrypted session flows include: Determine the IP address and port number of each communicating pair based on the five-tuple information of each data packet; Based on the five-tuple information of each data packet, the data packets related to the IP addresses and port numbers interacting between the two communicating parties are divided into the corresponding encrypted session traffic; S3. Extracting multimodal features of each encrypted session traffic according to the data packets of the encrypted session traffic, including metadata features, encrypted payload byte features, and burst transmission features; S4. Based on the multimodal features of several encrypted session flows and the categories corresponding to the original PCAP files to which each encrypted session flow belongs, construct a dataset including multiple samples, each sample including input data constructed based on the multimodal features of each encrypted session flow and a label indicating the category to which the encrypted session flow belongs.

2. The method according to claim 1, characterized in that In S3, the metadata features include multiple time series data, and the extraction methods of the multiple time series data include: Obtain a preset number of data packets from each encrypted session flow in chronological order; Extracting metadata for each of the acquired preset number of data packets, including the byte length and arrival time interval of the data packet, where the arrival time interval is determined based on the difference between the arrival time of the data packet and the arrival time of the immediately preceding adjacent data packet; Multiple time series data are obtained based on the metadata of all data packets, including the byte length and arrival time interval of each data packet, to obtain a time series data.

3. The method according to claim 1, characterized in that In S3, the method for extracting the encrypted payload byte feature in the multimodal feature includes: Extracting a preset number of bytes from each encrypted session flow, and all the extracted bytes do not include bytes at the packet header and bytes at the packet tail; A preset number of bytes are constructed into a payload byte image of a preset size, and the byte image is used as an encrypted payload byte feature, wherein the numerical values ​​of the preset number of bytes are used as the values ​​of each pixel in the byte image.

4. The method according to claim 1, wherein In S3, the method for extracting the burst transmission feature from the multimodal feature includes: Obtain a preset number of data packets from each encrypted session in chronological order, and count the number of burst transmission processes present in all the acquired data packets, where each burst transmission process represents a continuous transmission process of multiple data packets in the same communication direction; Counting the number of bytes transmitted in each of the multiple burst transmission processes, where the number of bytes transmitted in each burst transmission process is the sum of the number of bytes of all data packets transmitted in the burst transmission process; Counting the time intervals corresponding to the plurality of burst transmission processes, where the time interval corresponding to each burst transmission process is the difference between the end time of the burst transmission process and the end time of the previous adjacent burst transmission process; generating a two-dimensional joint distribution graph according to the number of bytes transmitted in each of the plurality of burst transmission processes and the corresponding time intervals, wherein the two-dimensional joint distribution graph includes a plurality of value pairs, each value pair recording the number of bytes transmitted in a burst transmission process and the corresponding time interval; The two-dimensional joint distribution graph is mapped into a burst distribution image of a preset size, and the distribution graph is used as a burst transmission feature.

5. The method according to any one of claims 1 to 4, characterized in that In S4, the multimodal features of an encrypted session flow are used to construct the input data of a sample. The construction method includes: The metadata feature of an encrypted session flow is used as the time feature, and the encrypted payload byte feature and burst transmission feature of the encrypted session flow are combined to obtain the spatial feature of the flow; Construct sample input data, which includes the temporal and spatial features of the encrypted session traffic.

6. The method according to any one of claims 1 to 4, characterized in that In said S1, the traffic classification task is an application traffic identification task, a business traffic identification task or a malicious traffic identification task, wherein, The various categories that need to be identified in the application traffic identification task include traffic categories corresponding to various applications; The various categories that need to be identified in the business traffic identification task include traffic categories corresponding to various businesses; The malicious traffic identification task needs to identify multiple categories including malicious traffic categories and normal traffic categories.

7. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 6.

8. An electronic device, characterized in that: include: one or more processors; as well as a memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 6 by executing the executable instructions.

Images