Data set construction method for encrypted network traffic classification

Abstract

The invention provides a data set construction method for encrypted network traffic classification, which comprises the following steps: acquiring encrypted traffic data from a network according to various categories required to be identified by a traffic classification task to obtain traffic data files including a plurality of PCAP files under each category; determining a plurality of encrypted session flows based on the flow data file, including one or more encrypted session flows contained in each PCAP file, each session flow including a plurality of data packets generated during interaction of two communication parties; according to the data packet of each session flow, multi-modal features of the session flow are extracted, and the multi-modal features comprise metadata features, encrypted payload byte features and burst transmission features; and according to the multi-modal features of the plurality of encrypted session flows and the category corresponding to the PCAP file to which each encrypted session flow belongs, a data set comprising a plurality of samples is constructed, and each sample comprises input data constructed according to the multi-modal features of each session flow and a label indicating the category to which the session flow belongs.

Description

Technical Field

[0001] The present invention relates to the field of network security, in particular to the field of encrypted network traffic identification, and more particularly to a data set construction method for encrypted network traffic classification. Background Art

[0002] With the rapid development of network technology, network applications emerge in an endless stream, and network security has also become a focus of attention. In the field of network security, network traffic identification is crucial, especially in distinguishing various forms of encrypted traffic, including protocols, applications, services, and malicious flows. However, with the widespread demand for protecting data transmission and protecting user privacy, the application scope of encryption technology has also expanded, and the encrypted traffic in the network has also soared. Although encrypted traffic protects user privacy, it also provides opportunities for designers of malicious traffic. Therefore, in order to meet the challenges of traffic encryption, it is very necessary to design a more robust encrypted network traffic classification method.

[0003] There are many methods for network traffic classification. Methods based on port numbers and methods based on deep packet inspection (DPI) are both rule-based methods. From the perspective of usability, after traffic encryption, some traditional rule-based traffic identification methods are no longer applicable due to port anonymity and difficulty in feature extraction, such as references [1] and [2]. From the perspective of security, the principles of these traditional methods are very clear, which also makes it easier for attackers to design more "real" malicious attack traffic. Some methods try to use the handshake information of the Transport Layer Security (TLS) 1.2 version to identify traffic, but with the widespread use of the encryption protocol TLS 1.3 version, some handshake information is also invisible.

[0004] In recent years, more work has begun to explore what features can be used for network traffic classification after traffic encryption. Because deep learning can directly extract and identify features from raw encrypted data, thus providing a high-precision end-to-end solution, more and more encrypted traffic identification methods based on deep learning have been proposed, such as references [3], [4], [5], and [6]. In addition, there are some other deep learning methods as follows:

[0005] For example, in the reference [7] proposed by Lotfollahi M et al., the Deep Packet method based on deep learning can handle the traffic characterization of network traffic classification into major categories (such as FTP and P2P), as well as application identification that requires end-user application identification (such as BitTorrent and Skype). It can not only identify encrypted traffic, but also distinguish between VPN network traffic and non-VPN network traffic. However, due to its relatively simple model design, although Deep Packet uses deep learning methods to achieve the goal of encrypted traffic identification, this method performs poorly in Tor traffic classification, so it still needs great improvement.

[0006] For example, the reference [8] proposed by C. Liu et al. is an end-to-end flow sequence network FS-Net for encrypted traffic identification. It uses packet length sequence as the network input and automatically realizes feature extraction and output. This method promotes feature learning through reconstruction mechanism. By keeping the reconstructed sequence and the original flow sequence as similar as possible, the generated features can contain more identification information to improve the recognition ability. However, FS-Net only uses the data packet length sequence as a traffic feature, which is easily affected by traffic camouflage techniques such as packet loss and disorder, and has poor robustness.

[0007] For example, the reference [9] proposed by T. Shapira and Y. Shavitt is a FlowPic method. This method utilizes the time-related and packet size-related characteristics of the flow to convert basic traffic data into intuitive FlowPic images, and then uses the known image classification deep learning technology CNN to identify traffic categories (such as browsing, chatting, video traffic categories, etc.). This method uses a single feature and is still affected by traffic camouflage technologies such as packet loss and disorder, and has poor anti-interference ability.

[0008] Considering that metadata features such as packet length sequence and packet arrival time interval sequence may be affected by traffic camouflage, some studies have found that the plaintext information of network flows is not completely randomized, but still has some differences. For example, reference

[10] shows that the plaintext information of network flows contains some semantic information and can also be used to classify network traffic. Therefore, some methods have emerged that use natural language processing models to learn the semantic information of the encrypted payload to identify encrypted traffic.

[0009] At the same time, in deep learning, the quality and quantity of input samples are also crucial to the model effect. However, models built by supervised learning methods such as CNN and RNN are highly dependent on labeled data. However, labeled data is not only small in size and complex in manual labeling, but may also have labeling errors. As a result, although these supervised learning methods can achieve good results on the training set, they are not well deployed in real network scenarios. To this end, combined with the characteristics of the encrypted payload information mentioned above, such as the reference

[11] proposed by Lin X et al., which is an encrypted traffic identification method based on the natural language processing model BERT, ET-BERT uses natural language processing methods to process and learn encrypted payload information, and performs unsupervised pre-training methods. However, encrypted traffic is very different from natural language and does not contain clear semantic information. ET-BERT only uses encrypted payload byte information, and the learned semantic features are weak, and the encrypted traffic classification effect is not ideal.

[0010] Therefore, when classifying encrypted network traffic, it is very difficult to extract features based on the existing rule-based methods, which makes it no longer suitable for extracting feature data for encrypted traffic. The encrypted information obtained by deep learning has the problems of large amount of data required and limited and single learnable feature information, which leads to poor classification effect and low robustness of the encrypted traffic classification model.

[0011] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. The relevant information is submitted and disclosed together with the present invention solution. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art.

[0012] References are as follows:

[0013] [1]VF Taylor, R. Spolaor, M. Conti, and I. Martinovic, "Robustsmartphone app identification via encrypted network traffic analysis," IEEETransactions on Information Forensics and Security, vol. 13, no. 1, Jan.2018, pp. 63-78.

[0014] [2] Y. Chen, T. Zang, Y. Zhang, Y. Zhou, and Y. Wang, “Rethinkingencrypted traffic classification: A multi-attribute associated fingerprintapproach,” in 27th IEEE International Conference on Network Protocols, ICNP2019, Chicago, IL, USA, October 8-10, 2019. IEEE, 2019, pp. 1–11.

[22] J. Che

[0015] [3] T. Ongun, T. Sakharaov, S. Boboila, A. Oprea, and T. Eliassi-Rad,“On designing machine learning models for malicious network trafficclassification,” CoRR, vol. abs / 1907.04846, 2019.

[0016] [4] Rezaei S, Liu X. Deep Learning for Encrypted TrafficClassification: An Overview[J]. IEEE Communications Magazine, 2019, 57(5):76-81.

[0017] [5] Lan J , Liu X , Li B , et al. DarknetSec: A novel self-attentivedeep learning method for darknet traffic classification and applicationidentification[J]. Computers & Security, 2022, 116(2):102663.

[0018] [6]J. Guo et al., "Global-Aware Prototypical Network for Few-ShotEncrypted Traffic Classification," 2022 IFIP Networking Conference (IFIPNetworking), Catania, Italy, 2022, pp. 1-9, doi: 10.23919 / IFIPNetworking55013.2022.9829771.

[0019] [7] Liu C, He L, Xiong G, et al. Fs-net: A flow sequence network forencrypted traffic classification[C] / / IEEE INFOCOM 2019-IEEE Conference OnComputer Communications. IEEE, 2019: 1171-1179.

[0020] [8] Lotfollahi M, Zade R S H, Siavoshani M J, et al. Deep Packet: ANovel Approach For Encrypted Traffic Classification Using Deep Learning[J].Soft Computing, 2017.

[0021] [9] T. Shapira and Y. Shavitt, "FlowPic: A Generic Representation forEncrypted Traffic Classification and Applications Identification," in IEEETransactions on Network and Service Management, vol. 18, no. 2, pp. 1218-1232, June 2021, doi: 10.1109 / TNSM.2021.3071441.

[0022]

[10] Xinjie Lin, Gang Xiong, Gaopeng Gou, Zhen Li, Junzheng Shi, JingYu: ETBERT: A Contextualized Datagram Representation with PretrainingTransformers for Encrypted Traffic Classification. WWW 2022.

[0023]

[11] Lin X, Xiong G, Gou G, et al. Et-bert: A contextualized datagram representation with pre-training transformers for encrypted traffic classification[C] / / Proceedings of the ACM Web Conference 2022. 2022: 633-642. Summary of the invention

[0024] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a data set construction method for encrypted network traffic classification.

[0025] The objective of the present invention is achieved through the following technical solutions:

[0026] According to a first aspect of the present invention, a method for constructing a data set for classifying encrypted network traffic is provided, comprising: S1, collecting encrypted traffic data from a network according to multiple categories required to be identified in a traffic classification task, and obtaining a traffic data file, which includes multiple original PCAP files corresponding to each of the multiple categories and used to record the encrypted traffic data; S2, determining a number of encrypted session traffic based on the traffic data file, including determining one or more encrypted session traffic contained in each original PCAP file according to the original PCAP file, each encrypted session traffic including a number of data packets generated during the interaction between two communicating parties; S3, extracting multimodal features of each encrypted session traffic according to the data packets of the encrypted session traffic, including metadata features, encrypted payload byte features, and burst transmission features; S4, constructing a data set including a number of samples according to the multimodal features of the several encrypted session traffic and the category corresponding to the original PCAP file to which each encrypted session traffic belongs, each sample including input data constructed according to the multimodal features of each encrypted session traffic and a label indicating the category to which the encrypted session traffic belongs.

[0027] In some embodiments of the present invention, in S2, determining one or more encrypted session flows contained in each original PCAP file according to the PCAP file includes: extracting original information from the original PCAP file, which includes multiple data packets and header fields of each data packet; determining one or more encrypted session flows in the form of sessions according to the header fields of each data packet, and dividing the multiple data packets into the corresponding encrypted session flows, wherein the encrypted session flows in the form of sessions include bidirectional flows formed during bidirectional communication between the communicating parties.

[0028] In some embodiments of the present invention, the packet header field includes five-tuple information consisting of a source IP address, a destination IP address, a source port number, a destination port number and a transport layer protocol, and the method of dividing multiple data packets into the encrypted session traffic to which they belong includes: determining the IP address and port number for interaction between each pair of communicating parties among multiple pairs of communicating parties based on the five-tuple information of each data packet; and dividing the data packets related to the IP address and port number for interaction between one pair of communicating parties into the corresponding encrypted session traffic based on the five-tuple information of each data packet.

[0029] In some embodiments of the present invention, in S3, the metadata features include multiple time series data, and the method of extracting the multiple time series data includes: obtaining a preset number of data packets in each encrypted session traffic in chronological order; extracting the metadata of each data packet in the preset number of data packets obtained, including the byte length and arrival time interval of the data packet, and the arrival time interval is determined according to the difference between the arrival time of the data packet and the arrival time of the adjacent data packet at the previous time; obtaining multiple time series data based on the metadata of all the obtained data packets, including splicing the byte length and arrival time interval of each data packet to obtain one time series data.

[0030] In some embodiments of the present invention, in S3, the method for extracting the encrypted payload byte feature in the multimodal feature includes: extracting a preset number of bytes in each encrypted session traffic, and all the extracted bytes do not include bytes in the packet header and bytes in the packet tail; constructing the preset number of bytes into a payload byte image of a preset size, and using the byte image as the encrypted payload byte feature, wherein the numerical values ​​of the preset number of bytes are used as the values ​​of each pixel in the byte image.

[0031] In some embodiments of the present invention, in S3, the method for extracting the burst transmission feature in the multimodal feature includes: obtaining a preset number of data packets in each encrypted session traffic in chronological order, and counting multiple burst transmission processes existing in all the obtained data packets, each burst transmission process represents a continuous transmission process of multiple data packets in the same communication direction; counting the number of bytes transmitted by each of the multiple burst transmission processes, the number of bytes transmitted by each burst transmission process is the sum of the number of bytes of all data packets transmitted by the burst transmission process; counting the time intervals corresponding to each of the multiple burst transmission processes, the time interval corresponding to each burst transmission process is the difference between the end time of the burst transmission process and the end time of the previous adjacent burst transmission process; generating a two-dimensional joint distribution graph according to the number of bytes transmitted by each of the multiple burst transmission processes and the corresponding time intervals, which includes multiple value pairs, each value pair records the number of bytes transmitted by a burst transmission process and the corresponding time interval; mapping the two-dimensional joint distribution graph into a burst distribution image of a preset size, and using the distribution graph as a burst transmission feature.

[0032] In some embodiments of the present invention, in S4, the multimodal features of an encrypted session flow are used to construct the input data of a sample, and the construction method includes: using the metadata features of an encrypted session flow as the time features, splicing the encrypted payload byte features and burst transmission features of the encrypted session flow, and obtaining the spatial features of the flow; constructing the input data of the sample, which includes the time features and spatial features of the encrypted session flow.

[0033] In some embodiments of the present invention, in S1, the traffic classification task is an application traffic identification task, a business traffic identification task or a malicious traffic identification task, wherein the multiple categories required to be identified in the application traffic identification task include traffic categories corresponding to multiple applications; the multiple categories required to be identified in the business traffic identification task include traffic categories corresponding to multiple businesses; and the multiple categories required to be identified in the malicious traffic identification task include malicious traffic categories and normal traffic categories.

[0034] According to a second aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method described in the first aspect of the present invention by executing the executable instructions.

[0035] Compared with the prior art, the advantages of the present invention are:

[0036] The method of the present invention collects multiple original PCAP files for each traffic category to ensure the integrity and diversity of samples in the data set; secondly, each encrypted session traffic includes the interactive information of the communicating parties, which expresses more information and is more conducive to subsequent feature extraction and traffic classification; then, the multimodal features of each encrypted session traffic are extracted, which include three features: metadata features, encrypted payload byte features, and burst transmission features, so that the data set constructed by the present invention includes multi-dimensional and rich traffic feature information; finally, each sample in the data set includes input data constructed according to the multimodal features of each encrypted session traffic, that is, includes multi-dimensional encrypted traffic features, which is more conducive to the model learning more traffic features and semantic information when applied to the training of a deep learning model, so as to improve the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0038] Figure 1 The figure is a flowchart of a method for constructing a data set according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] As mentioned in the background technology section, when classifying encrypted network traffic, existing rule-based methods are very difficult to extract features, resulting in them no longer being applicable to extract feature data for encrypted traffic. The encrypted information learned through deep learning requires a large amount of data and the learnable feature information is limited and single, which in turn leads to poor classification effect and low robustness of the encrypted traffic classification model.

[0041] To solve the above problems, the inventors conducted research and analysis from multiple angles and found that: 1) Although the rule-based method extracts the characteristics of encrypted traffic, namely metadata characteristics, it will be affected by traffic camouflage, but its role cannot be completely denied; 2) Although the semantic information of the encrypted payload bytes is relatively weak, it cannot deny its role; 3) Burst transmission reflects the behavioral information of traffic. In order to reduce the burden on the CPU, some applications will cache the data packets to be sent first, and then send them together when a certain number of caches are reached, so the application layer always tends to send or receive as much data as possible at a time, thus forming a burst transmission. The burst transmission of different types of traffic is also different, so the burst feature can also be used as an important feature for identifying encrypted network traffic.

[0042] Based on the above research and analysis, according to an embodiment of the present invention, a method for constructing a data set for encrypted network traffic classification is provided. First, for each traffic category required to be identified in the traffic classification task, the method of the present invention collects multiple original network packet capture format (Packet Capture, referred to as PCAP) files for recording encrypted traffic to ensure the integrity and diversity of samples in the data set; secondly, the encrypted session traffic in the form of a session contained in each original PCAP file is determined, and each encrypted session traffic includes the interactive information of the communicating parties, which expresses more information and is more conducive to subsequent feature extraction and traffic classification; then, the multimodal features of each encrypted session traffic are extracted, which include metadata features, encrypted payload byte features, and burst transmission features, so that the data set constructed by the present invention includes multi-dimensional and rich traffic feature information; finally, each sample in the data set includes input data constructed according to the multimodal features of each encrypted session traffic and a label indicating the category to which the encrypted session traffic belongs. The constructed dataset includes multi-dimensional encrypted traffic features. When applied to the training of deep learning models, compared with the training and learning method that only uses metadata features or encrypted payload byte features, the dataset constructed by the present invention is more conducive to the model learning more traffic features and semantic information to improve the robustness of the model.

[0043] In general, according to one embodiment of the present invention, the construction of the data set of the present invention can be seen in Figure 1 , which is a flowchart of a method for constructing a data set. It includes steps S1, S2, S3 and S4. After the data set is constructed, the data set can be used to train a model for classifying encrypted network traffic to obtain a model that can classify encrypted network traffic. In order to better understand the present invention, the steps of the method for constructing a data set and the application of the data set of the present invention are described in detail below in conjunction with specific embodiments.

[0044] 1. Dataset Construction Method

[0045] In step S1, encrypted traffic data is collected from the network according to the multiple categories required to be identified in the traffic classification task to obtain a traffic data file, which includes multiple original PCAP files corresponding to each of the multiple categories for recording encrypted traffic data.

[0046] According to one embodiment of the present invention, the traffic classification task is an application traffic identification task, a business traffic identification task or a malicious traffic identification task, etc., and the present invention is not limited to this. Among them, the multiple categories that need to be identified in the application traffic identification task include traffic categories corresponding to multiple applications, the multiple categories that need to be identified in the business traffic identification task include traffic categories corresponding to multiple businesses, and the multiple categories that need to be identified in the malicious traffic identification task include malicious traffic categories and normal traffic categories. Schematically, the traffic categories corresponding to multiple applications may include WeChat application traffic categories, Taobao application traffic categories, Douyin application traffic categories, Meituan application traffic categories, etc.; the traffic categories corresponding to multiple businesses may include live broadcast traffic categories, on-demand traffic categories, voice traffic categories, download traffic categories, game traffic categories, music traffic categories, etc.

[0047] According to one embodiment of the present invention, taking the task of application traffic identification as an example, tools such as Wireshark or Tshark can be used to capture traffic packets under various applications in the network and save them as original PCAP files. The original PCAP file of the application traffic records the application to which the original PCAP file belongs when the tool is used to capture the packet. This is to distinguish the application traffic category to which the encrypted network traffic recorded in the file belongs.

[0048] In step S2, a number of encrypted session flows are determined based on the flow data file, including determining one or more encrypted session flows contained in each original PCAP file according to the PCAP file, and each encrypted session flow includes a number of data packets generated during the interaction between the communicating parties.

[0049] According to an embodiment of the present invention, in step S2, the method of determining one or more encrypted session flows contained in each original PCAP file includes the following steps S21 and S22:

[0050] Step S21: extracting original information from the original PCAP file, which includes multiple data packets and the packet header field of each data packet.

[0051] According to one embodiment of the present invention, the content of each original PCAP file is read one by one to extract the original information of each original PCAP file. The original information includes all data packets contained in the PCAP file and each data packet information, and each data packet information includes the source IP address, the destination IP address, the source port number, the destination port number, the transport layer protocol, the byte length of the data packet, and the time when the data packet arrives. The packet header field includes a five-tuple information consisting of the source IP address, the destination IP address, the source port number, the destination port number, and the transport layer protocol.

[0052] Step S22: Determine one or more encrypted session flows in the form of sessions according to the header fields of each data packet, and divide the multiple data packets into the corresponding encrypted session flows, wherein the encrypted session flows in the form of sessions include bidirectional flows formed during bidirectional communication between the communicating parties.

[0053] According to an embodiment of the present invention, the encrypted session traffic includes bidirectional traffic, and all data packets of each traffic in the bidirectional traffic have the same five-tuple information, and the encrypted session traffic includes all data packets of the bidirectional traffic. Schematically, if the two communicating parties are Party A and Party B, the bidirectional traffic includes a traffic corresponding to data transmitted from Party A to Party B and a traffic corresponding to data transmitted from Party B to Party A. The technical solution of this embodiment can at least achieve the following beneficial technical effects: because the session traffic includes bidirectional traffic, which contains the interactive information of the communicating parties, more information can be obtained, which is more conducive to traffic classification.

[0054] According to one embodiment of the present invention, the method of dividing multiple data packets into the encrypted session traffic to which they belong includes: determining the IP address and port number of each pair of communicating parties interacting in multiple pairs according to the five-tuple information of each data packet; dividing the data packets related to the IP address and port number of a pair of communicating parties interacting into the corresponding encrypted session traffic according to the five-tuple information of each data packet. That is, the five-tuple information of all data packets in the encrypted session traffic includes the IP address and port number of a pair of communicating parties interacting. And after the source IP address and the destination IP address in the five-tuple information of a data packet in the encrypted session traffic are interchanged, and the source port number and the destination port number are interchanged, the data packet still belongs to the encrypted session traffic.

[0055] In step S3, according to the data packets of each encrypted session flow, the multimodal features of the encrypted session flow are extracted, which include metadata features, encrypted payload byte features and burst transmission features.

[0056] According to an embodiment of the present invention, the following describes the extraction methods of metadata features, encrypted payload byte features, and burst transmission features respectively:

[0057] 1) Extraction of metadata features

[0058] According to one embodiment of the present invention, the metadata feature includes multiple time series data, and the method for extracting the multiple time series data includes: obtaining a preset number of data packets in each encrypted session traffic in chronological order; extracting metadata of each data packet in the preset number of data packets obtained, including the byte length and arrival time interval of the data packet, the byte length of each data packet includes the byte length occupied by all data in the packet, and the arrival time interval is determined based on the difference between the arrival time of the data packet and the arrival time of the adjacent data packet at the previous time; obtaining multiple time series data based on the metadata of all the obtained data packets, including splicing the byte length and arrival time interval of each data packet to obtain one time series data.

[0059] According to an embodiment of the present invention, n packets transmitted previously in each encrypted session flow are obtained in chronological order. It should be noted that when extracting the three features of metadata features, encrypted payload byte features, and burst transmission features, the packets used and the total number of packets used are the same, which can be 500 packets transmitted previously, 1000 packets transmitted previously, or 1200 packets transmitted previously, etc. for each encrypted session flow, and the present invention does not limit the value of n.

[0060] According to one embodiment of the present invention, the metadata of each of the n acquired data packets is extracted, including counting the byte length and arrival time interval of the n data packets transmitted before each encrypted session traffic. When n=1000, the byte length of each of the 1000 data packets transmitted before each encrypted session traffic is counted to obtain a byte sequence containing a 1000-byte length, and the arrival time interval of each of the first 1000 data packets of the encrypted session traffic, to obtain a time series containing 1000 arrival time intervals, and the byte sequence and the time series are spliced ​​one by one to form a metadata feature containing 1000 time series data and each time series data dimension is 2. The present invention extracts time series data as metadata features to achieve the analysis of traffic types from a time perspective.

[0061] 2) Extraction of encrypted payload byte features

[0062] According to one embodiment of the present invention, the method for extracting the encrypted payload byte feature in the multimodal feature includes: extracting a preset number of bytes from each encrypted session flow, and all the extracted bytes do not include the bytes of the packet header and the bytes of the packet tail; wherein the partially extracted bytes are encrypted payload bytes, and the encrypted payload bytes refer to the actual transmitted data located in the middle part of the packet after removing the byte information of the packet header and the tail in network communication. The extracted preset number of bytes are constructed into a payload byte image of a preset size, and the byte image is used as the encrypted payload byte feature, wherein the values ​​of the preset number of bytes are used as the values ​​of each pixel in the byte image.

[0063] According to one embodiment of the present invention, the 1000 data packets transmitted before each encrypted session traffic are obtained, and after removing the header and tail bytes of each data packet, bytes are collected from the 1000 data packets from which the header and tail bytes are removed. Take the construction of a payload byte image with a preset size of 256×256×1 (length×width×number of channels, the same below) as an example to illustrate: 65536 bytes are collected from the 1000 data packets from which the header and tail bytes are removed. Among them, if the bytes in the 1000 data packets from which the header and tail bytes are removed are too long, the bytes of the following data packets are trimmed and only the first 65536 bytes are taken; if it is less than 65536 bytes, 0 is added to all bytes of the 1000 data packets from which the header and tail bytes are removed until 65536 bytes are met. Using 65536 bytes, and reconstructing an image with a height of 256 and a width of 256 in a way that one byte corresponds to one pixel of the image, a payload byte image with a size of 256×256×1 is generated. Since the size of each byte is a value of 0-255, the pixels of the payload byte image are values ​​of 0-255. The payload byte image is used as the encrypted payload byte feature. If a payload byte image with a preset size of 128×128×1 is constructed, 16384 bytes need to be collected from the 1000 data packets with the head and tail bytes removed to construct the image according to the above implementation method.

[0064] 3) Extraction of Burst Transmission Features

[0065] According to an embodiment of the present invention, a method for extracting a burst transmission feature includes steps a1, a2, a3, a4 and a5:

[0066] Step a1: Obtain a preset number of data packets in each encrypted session flow in chronological order, and count multiple burst transmission processes existing in all the acquired data packets, where each burst transmission process represents a continuous transmission process of multiple data packets in the same communication direction.

[0067] According to an embodiment of the present invention, burst transmission means the continuous transmission of multiple data packets in the same communication direction, and each encrypted session traffic is a bidirectional traffic. For example, in the encrypted session traffic generated when A and B communicate with each other, the 1000 data packets previously transmitted by the encrypted session traffic are obtained, and the transmission from A to B and the transmission from B to A are two directions. If the first to the fifth data packets in the 1000 data packets obtained are all continuously transmitted from A to B, then the process of transmitting the first to the fifth data packets constitutes a burst transmission process. In this way, the number of burst transmission processes existing in the 1000 data packets obtained is counted.

[0068] Step a2: Count the number of bytes transmitted by each of the multiple burst transmission processes. The number of bytes transmitted by each burst transmission process is the sum of the number of bytes of all data packets transmitted by the burst transmission process.

[0069] According to an embodiment of the present invention, if a burst transmission process includes 5 data packets transmitted continuously, the number of bytes of the 5 data packets is summed to obtain the number of bytes transmitted by the burst transmission process. When counting, all burst transmission processes are counted in time order to obtain the number of bytes transmitted by each of the multiple burst transmission processes. Schematically, taking 1000 data packets as an example, the process of continuously transmitting the 1st to 8th data packets in the same communication direction is the first burst transmission process, the process of continuously transmitting the 9th to 15th data packets in the same communication direction is the second burst transmission process, and the process of continuously transmitting the 16th to 22nd data packets in the same communication direction is the third burst transmission process, ..., if the number of bytes transmitted by the first, second, and third burst transmission processes is counted in turn as 64, 128, 32, ..., the number of bytes transmitted by each of the multiple burst transmission processes in the 1000 data packets is obtained.

[0070] Step a3: Counting the time intervals corresponding to the multiple burst transmission processes, where the time interval corresponding to each burst transmission process is the difference between the end time of the burst transmission process and the end time of the previous adjacent burst transmission process.

[0071] According to an embodiment of the present invention, schematically, in the first, second, and third burst transmission processes given by the example of the above embodiment, the time interval of the first burst transmission process is recorded as the difference between the end time of the burst transmission process and 0, the time interval of the second burst transmission process is the difference between the end time of the burst transmission process and the end time of the first burst transmission process, and the time interval of the third burst transmission process is the difference between the end time of the burst transmission process and the end time of the second burst transmission process, so as to calculate the time intervals corresponding to the multiple burst transmission processes.

[0072] Step a4: Generate a two-dimensional joint distribution graph according to the number of bytes transmitted in each of the multiple burst transmission processes and the corresponding time intervals, which includes multiple value pairs, each value pair recording the number of bytes transmitted in a burst transmission process and the corresponding time interval.

[0073] According to one embodiment of the present invention, the number of bytes transmitted by each of the multiple burst transmission processes is normalized, and the time intervals corresponding to each of the multiple burst transmission processes are normalized to generate a two-dimensional joint distribution picture (two-dimensional graph) of the number of bytes and time intervals of the multiple burst transmission processes. Normalization is to map all values ​​to the interval [0,1]. Each value pair in the two-dimensional joint distribution graph records the normalized number of bytes corresponding to a burst transmission process and the corresponding normalized time interval in a two-dimensional form. Schematically, assuming that the horizontal axis of the graph is the x-axis and the vertical axis is the y-axis, then the x-axis refers to a deduplicated ordered sequence formed by the normalized number of bytes of each of the multiple burst transmission processes, and the y-axis refers to a deduplicated ordered sequence formed by the normalized time intervals of each of the multiple burst transmission processes. The de-duplicated ordered sequence means a re-ordered sequence after removing repeated values. For example, the original sequence formed by the normalized number of bytes of multiple burst transmission processes is [0.1, 0.5, 0.2, 0.8, 0.2], and after de-duplication and sorting, it becomes [0.1, 0.2, 0.5, 0.8]. Then any point on this two-dimensional graph corresponds to a two-dimensional value pair. For example, in the value pair (Xi, Yi) corresponding to the i-th point, Xi is the normalized number of bytes of the burst transmission process corresponding to the i-th point, and Yi is the normalized time interval of the burst transmission process corresponding to the i-th point.

[0074] Step a5: Mapping the two-dimensional joint distribution graph into a burst distribution image of a preset size, and using the distribution graph as a burst transmission feature.

[0075] According to one embodiment of the present invention, the two-dimensional joint distribution graph can be proportionally mapped to a height of 256 and a width of 256. Thus, a burst distribution image is obtained, the image size is 256×256×1, 1 is the number of channels, and the distribution graph is used as a burst transmission feature. In addition, the preset sizes of the payload byte image and the burst distribution image are generally set to be the same, which can be 256×256×1, 128×128×1, or 64×64×1, and the present invention is not limited to this. This is to facilitate the subsequent construction of spatial features based on the payload byte features and the burst transmission features.

[0076] In step S4, a data set including multiple samples is constructed according to the multimodal features of several encrypted session flows and the category corresponding to the original PCAP file to which each encrypted session flow belongs, and each sample includes input data constructed according to the multimodal features of each encrypted session flow and a label indicating the category to which the encrypted session flow belongs.

[0077] According to one embodiment of the present invention, the multimodal features of an encrypted session flow are used to construct the input data of a sample, and the construction method includes: using the metadata features of an encrypted session flow as the time features, splicing the encrypted payload byte features and burst transmission features of the encrypted session flow, and obtaining the spatial features of the flow; constructing the input data of the sample, which includes the time features and spatial features of the encrypted session flow.

[0078] According to an embodiment of the present invention, the metadata feature is used as a time feature, the encrypted payload byte feature is used as a channel of the image, and the burst transmission feature is extracted as another channel of the image, and a dual-channel image is jointly generated as a spatial feature. Schematically, the metadata feature of each encrypted session traffic is used as a time feature, and the 1000 data packets transmitted before each encrypted session traffic are obtained as an example. The time feature is a multi-channel time series with a length of 1000 and a channel number of 2. The obtained 256×256×1 payload byte image and 256×256×1 burst distribution image are spliced ​​in the channel dimension, that is, a picture with a height and width of 256 and a channel number of 2 is obtained (the picture dimension is recorded as 256×256×2), which is the spatial feature of the sample input data. Thus, the spatial feature input and time feature input of each sample can be obtained. The technical solution of this embodiment can at least achieve the following beneficial technical effects: the input data of the sample of the present invention includes multimodal traffic feature information, which is conducive to the model learning traffic features and semantic information from multiple dimensions to improve the traffic recognition performance of the model.

[0079] 2. Application of Dataset

[0080] According to one embodiment of the present invention, a model for classifying encrypted network traffic may be trained based on the data set obtained in the above embodiment, and the application implementation process of the data set is described as follows:

[0081] Implementation Method 1

[0082] According to one embodiment of the present invention, a training method for an encrypted network traffic classification model is provided, including a pre-training stage and a downstream task training stage, wherein the encrypted network traffic classification model includes a deep learning model and a classifier, the deep learning model is trained in the pre-training stage to obtain a pre-trained deep learning model, and the parameters of the pre-trained deep learning model are frozen in the downstream task training stage to train the classifier. The deep learning model can be formed by stacking Transformers or constructed by other deep neural networks, and the present invention is not limited to this.

[0083] According to one embodiment of the present invention, the process of training a deep learning model in the pre-training stage includes the following process:

[0084] 1) Based on the data set constructed by the method of the above embodiment, a first training set is constructed, including multiple positive samples and negative samples. The input data of each positive sample includes the time features and spatial features from the same encrypted session traffic, and the input data of each negative sample includes the time features and spatial features from different encrypted session traffic. The first label of each positive sample and negative sample is used to indicate whether the time features and spatial features of its input data match. The first label of the positive sample is marked as 1 to indicate a match, and the first label of the negative sample is marked as 0 to indicate a mismatch.

[0085] 2) Using the first training set to train the deep learning model, the temporal features and spatial features of the input data of each sample in the first training set are processed separately to obtain the temporal coding result and spatial coding result of each sample in the first training set, and the matching degree of the temporal features and spatial features of the sample is obtained according to the temporal coding result and spatial coding result of each sample in the first training set. The parameters of the deep learning model are updated with the optimization goal of minimizing the difference between the matching degree of each sample in the first training set and the first label of the sample to obtain the pre-trained deep learning model.

[0086] According to one embodiment of the present invention, the deep learning model includes a first deep neural network module and a second deep neural network module. The time features of the input data of each sample in the first training set are encoded by the first deep neural network module to obtain a time encoding result. The spatial features of the input data of each sample in the first training set are encoded by the second deep neural network module to obtain a spatial encoding result. The degree of matching between the time features and the spatial features can be obtained by calculating the similarity between the time encoding results and the spatial encoding results of each sample in the first training set. The parameters of the deep learning model are updated by maximizing the similarity between the time encoding results and the spatial encoding results of the positive samples and minimizing the similarity between the time encoding results and the spatial encoding results of the negative samples as the optimization goal to obtain a pre-trained deep learning model. The pre-training process enables the deep learning model to learn the deep semantic association between spatial features and time features.

[0087] According to one embodiment of the present invention, the process of training a classifier in the downstream task training phase includes the following process:

[0088] 3) Obtain a second training set constructed by the deep learning model pre-trained in the above embodiment and a data set constructed based on the method of the above embodiment, including multiple samples, each sample including input data constructed based on multimodal features extracted from each encrypted session traffic and a label indicating the traffic type to which the encrypted session traffic belongs, and the input data includes time characteristics and spatial characteristics of the same encrypted session traffic.

[0089] 4) Training the classifier using the second training set includes the following steps:

[0090] 4.1) Duplicate the time features of each sample in the second training set multiple times, and splice different traffic category attributes [CLS] in front of each time feature, and then splice with the randomly initialized soft prompt to obtain multiple pre-processed time features. Input the multiple pre-processed time features of each sample into the first deep neural network module of the pre-trained deep learning model for encoding, and obtain multiple time encoding results. Among them, the encrypted network traffic classification model needs to replicate the time features of each sample as many times as the number of categories it needs to identify.

[0091] 4.2) Input the spatial features of each sample in the second training set into the second deep neural network module of the pre-trained deep learning model for encoding, and obtain the spatial encoding result, and input the spatial encoding result into the classifier to obtain the model output result, which includes the probability value of the sample belonging to each traffic category. Among them, the probability value is obtained by calculating the similarity between the spatial encoding result and each time encoding result in multiple time encoding results, and the traffic category attribute corresponding to the time encoding result with the highest similarity to the spatial encoding result is used as the traffic category of the sample.

[0092] 4.3) Freeze the parameters of the pre-trained deep learning model and update the parameters of the classifier with the optimization goal of minimizing the difference between the model output and the label.

[0093] The technical solution of this implementation mode 1 achieves at least the following beneficial effects: in the pre-training stage, the same deep learning model can be adjusted for different downstream tasks, and in the downstream task training stage, only a few parameters need to be fine-tuned, thereby enhancing its applicability to a wide range of downstream tasks.

[0094] Implementation Method 2

[0095] According to another embodiment of the present invention, another training method for an encrypted network traffic classification model is provided, comprising: obtaining a data set constructed based on the method of the above embodiment, which includes multiple samples, each sample including input data constructed based on multimodal features extracted from each encrypted session traffic and a label indicating the traffic type to which the encrypted session traffic belongs; using the data set to train the encrypted network traffic classification model to learn to identify the traffic type to which the encrypted session traffic belongs based on the input data, obtain recognition results for each encrypted session traffic, and update the parameters of the model with the optimization goal of minimizing the difference between the recognition results and the labels.

[0096] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0097] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0098] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0099] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for constructing a dataset for encrypted network traffic classification, characterized in that: include: S1. Collect encrypted traffic data from the network according to the multiple categories required to be identified by the traffic classification task, and obtain a traffic data file, which includes multiple original PCAP files corresponding to each of the multiple categories for recording the encrypted traffic data; S2. Determine a plurality of encrypted session flows based on the flow data file, including determining one or more encrypted session flows contained in each original PCAP file according to the PCAP file, wherein each encrypted session flow includes a plurality of data packets generated during the interaction between the communicating parties; S3, extracting multimodal features of each encrypted session flow according to the data packet of the encrypted session flow, which includes metadata features, encrypted payload byte features and burst transmission features; S4. According to the multimodal features of several encrypted session flows and the category corresponding to the original PCAP file to which each encrypted session flow belongs, a data set including multiple samples is constructed, each sample including input data constructed according to the multimodal features of each encrypted session flow and a label indicating the category to which the encrypted session flow belongs.

2. The method according to claim 1, characterized in that: In S2, determining, according to each original PCAP file, one or more encrypted session flows contained in the PCAP file includes: Extracting original information from the original PCAP file, which includes a plurality of data packets and a header field of each data packet; According to the packet header fields of each data packet, one or more encrypted session flows in the form of a session are determined, and multiple data packets are divided into the corresponding encrypted session flows, wherein the encrypted session flows in the form of a session include bidirectional flows formed during bidirectional communication between two communicating parties.

3. The method according to claim 2, characterized in that The packet header field includes five-tuple information consisting of a source IP address, a destination IP address, a source port number, a destination port number, and a transport layer protocol. The method of dividing the multiple data packets into the encrypted session traffic includes: According to the five-tuple information of each data packet, determine the IP address and port number of each pair of communicating parties in multiple pairs of communicating parties; According to the five-tuple information of each data packet, the data packets related to the IP addresses and port numbers interacting between a pair of communicating parties are divided into the corresponding encrypted session traffic.

4. The method according to claim 1, characterized in that In S3, the metadata feature includes multiple time series data, and the extraction method of the multiple time series data includes: Obtain a preset number of data packets in each encrypted session flow in chronological order; Extracting metadata of each data packet from the acquired preset number of data packets, including the byte length and arrival time interval of the data packet, where the arrival time interval is determined according to the difference between the arrival time of the data packet and the arrival time of the previous adjacent data packet; Multiple time series data are obtained according to the metadata of all the acquired data packets, including the byte length and arrival time interval of each data packet, to obtain a time series data.

5. The method according to claim 1, characterized in that In S3, the method for extracting the encrypted payload byte feature in the multimodal feature includes: Extracting a preset number of bytes from each encrypted session flow, and all the extracted bytes do not include bytes at the header of the data packet and bytes at the tail of the data packet; A preset number of bytes are constructed into a payload byte image of a preset size, and the byte image is used as an encrypted payload byte feature, wherein the respective numerical values ​​of the preset number of bytes are used as the values ​​of each pixel in the byte image.

6. The method according to claim 2, characterized in that In S3, the method for extracting the burst transmission feature in the multimodal feature includes: Obtain a preset number of data packets in each encrypted session flow in chronological order, and count multiple burst transmission processes existing in all the acquired data packets, where each burst transmission process represents a continuous transmission process of multiple data packets in the same communication direction; Counting the number of bytes transmitted by each of the multiple burst transmission processes, the number of bytes transmitted by each burst transmission process is the sum of the number of bytes of all data packets transmitted by the burst transmission process; Counting the time intervals corresponding to the multiple burst transmission processes, where the time interval corresponding to each burst transmission process is the difference between the end time of the burst transmission process and the end time of the previous adjacent burst transmission process; Generate a two-dimensional joint distribution graph according to the number of bytes transmitted in each of the plurality of burst transmission processes and the corresponding time interval, the two-dimensional joint distribution graph comprising a plurality of value pairs, each value pair recording the number of bytes transmitted in a burst transmission process and the corresponding time interval; The two-dimensional joint distribution graph is mapped into a burst distribution image of a preset size, and the distribution graph is used as a burst transmission feature.

7. The method according to any one of claims 1 to 6, characterized in that: In S4, a multimodal feature of an encrypted session flow is used to construct input data of a sample, and the construction method includes: The metadata feature of an encrypted session flow is used as the time feature, and the encrypted payload byte feature and burst transmission feature of the encrypted session flow are concatenated to obtain the spatial feature of the flow; Construct sample input data, which includes the temporal characteristics and spatial characteristics of the encrypted session traffic.

8. The method according to any one of claims 1 to 6, characterized in that: In S1, the traffic classification task is an application traffic identification task, a business traffic identification task or a malicious traffic identification task, wherein: The various categories that need to be identified in the application traffic identification task include traffic categories corresponding to various applications; The various categories that need to be identified in the business traffic identification task include traffic categories corresponding to various businesses; The malicious traffic identification task needs to identify multiple categories including malicious traffic categories and normal traffic categories.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1-8 by executing the executable instructions.

Images