Encrypted traffic anomaly detection method based on high-performance traffic collection

Abstract

The invention provides an encrypted traffic anomaly detection method based on high-performance traffic collection, and the method specifically comprises the following steps: 1) achieving the capturing and collection of high-speed traffic through the high-performance traffic collection based on DPDK, and extracting traffic features by taking a flow as a unit, and carrying out the persistence of the traffic features in a traffic database; 2) providing a two-stage technology of automatic online encrypted traffic collection and sample construction and service classification model construction based on the service classification of the encrypted traffic to realize specific service classification for the encrypted traffic; and 3) proposing construction of a device-level traffic portrait to realize space-time granularity division of a traffic information prediction result, proposing various abnormal conditions and corresponding judgment strategies according to the device-level traffic in a real network environment, constructing a traffic anomaly detection model, and realizing timely early warning of the abnormal conditions. According to the method, the support service of the whole flow of flow collection, service classification and anomaly detection is realized, the flow behavior anomaly of each service device is dealt with in time, and the effectiveness and reliability of the operation of a network and an information system are ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of cyberspace security and relates to an encrypted traffic anomaly detection method based on high-performance traffic collection. Background Art

[0002] With the rapid development of network technology, the scale of the network continues to expand, making the Internet an indispensable part of human life. As a carrier of network information transmission, network traffic can be analyzed to understand business types, perceive abnormal behaviors, detect malicious attacks, predict link bandwidth, etc. Therefore, network traffic detection has become one of the most important tasks in the network field and an important prerequisite for network resource management, cyberspace security, and network service quality. Especially for cyberspace security, network traffic detection, as an effective means of network protection, can perceive abnormal traffic in the network and provide important support for various tasks. It has long been widely concerned by scholars.

[0003] However, with the continuous evolution of network encryption and obfuscation technologies, encrypting and obfuscating network traffic has become a common practice in the industry. After encryption and obfuscation, the traffic fingerprint, statistical characteristics, and traffic distribution of network traffic will change, making it increasingly difficult to achieve high-speed and accurate network traffic detection and identification. At the same time, various network attack methods on the Internet continue to evolve, and a large number of new network attacks have also appeared in cyberspace, which has had a huge impact on cyberspace security. Existing methods have many shortcomings in detecting unknown attacks. What's worse, with the support of encryption and obfuscation technology, malicious traffic can more easily evade existing network anomaly detection systems, posing a great threat to network security.

[0004] In order to achieve accurate and efficient encrypted traffic anomaly detection, domestic and foreign researchers usually use a variety of technical means, which can generally be divided into anomaly detection methods based on statistical analysis, detection methods based on feature engineering, detection methods based on traditional machine learning, and detection methods based on deep learning. However, the existing methods are not suitable for high-speed networks and large-scale real network environments. Whether it is traditional machine learning methods or the calculation methods of popular deep learning methods, they have encountered unprecedented limitations in network security detection, making the existing detection methods unable to meet the real-time and accuracy requirements of attack detection in large-scale high-speed real network environments. Specifically, encrypted traffic anomaly detection has the following problems: (1) The computational cost is high and it is difficult to apply to high-speed network environments; (2) The existing methods are not accurate enough; (3) They focus on traditional protocols and cannot be applied to complex new network protocol scenarios; (4) The model is too complex and is limited by hardware processing capabilities; (5) The analysis method and feature selection rely heavily on manual experience; (6) The methods in the experimental environment ignore the problems and characteristics in the actual network environment. Therefore, the present invention performs high-speed capture and collection of real traffic based on DPDK, and realizes efficient service classification, high-precision service classification and balanced service classification through multi-level encrypted traffic service classification, thereby providing a data basis for subsequent encrypted traffic anomaly detection, constructing a device-level traffic portrait, and based on various abnormal situations and corresponding evaluation strategies of device-level traffic in real network scenarios, a traffic anomaly detection model is constructed to realize encrypted traffic anomaly detection. Summary of the invention

[0005] In view of the various problems of existing encrypted traffic anomaly detection technologies, such as complex calculations, insufficient accuracy, over-reliance on experts, difficulty in adapting to complex new network protocols, and inability to apply to actual network environments, the present invention proposes a design and implementation scheme for encrypted traffic anomaly detection based on high-performance traffic collection. Compared with existing encrypted traffic anomaly detection, this scheme will comprehensively consider the relationship between business characteristics and anomaly features, aiming to obtain more comprehensive, efficient, and accurate encrypted traffic information, thereby achieving applicability to high-speed network environments. The present invention implements support services for the entire process of traffic collection-business classification-anomaly detection, responds to traffic behavior anomalies of service equipment in a timely manner, and ensures the effectiveness and reliability of network and information system operations. At the same time, an incremental self-evolution strategy is used to achieve dynamic updates of portraits and models, ensure the longevity of the life cycle of the anomaly detection function, and ensure continuous encrypted traffic supervision in real networks.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A design and implementation scheme for encrypted traffic anomaly detection based on high-performance traffic collection includes the following steps:

[0008] (1) Based on DPDK, high-performance traffic collection is performed to capture and collect high-speed traffic in network data centers. At the same time, the spatiotemporal features of the traffic are extracted based on flows (a collection of data packets with the same source IP address, sink IP address, source port, sink port, and protocol), and the extracted spatiotemporal features are persisted in the database.

[0009] (2) To meet the specific business classification requirements for encrypted traffic, a two-stage technology of automated online encrypted traffic collection and sample construction, and business classification model construction is proposed to realize system functions. The automated online encrypted traffic collection and sample construction part will sequentially perform automated sample collection and preliminary labeling, sample label construction, sample purification and feature extraction. Multi-level business classification based on heterogeneous features is divided into efficient business classification based on lightweight models, high-precision business classification based on deep learning models, and balanced business classification based on multi-model combination;

[0010] (3) Based on high-performance traffic collection and multi-level encrypted traffic service classification, service-oriented anomaly detection is realized, and the construction of device-level traffic portraits is proposed. From the perspective of system database construction, the spatiotemporal granularity of traffic information and prediction results is realized, and the spatiotemporal division of service classification is realized. Based on the device-level traffic in the network scenario, a variety of abnormal situations and corresponding evaluation strategies are proposed, and a traffic anomaly detection model is built based on this to realize the anomaly detection function. At the same time, in response to the concept drift problem in the field of traffic analysis, a life cycle maintenance method for device-level traffic portraits and service classification models is designed based on the self-evolution strategy.

[0011] Furthermore, the step (1) specifically includes the following sub-steps:

[0012] (1.1) The present invention designs a high-performance traffic collection system based on DPDK to capture and collect high-speed traffic in network data centers, and extracts the spatiotemporal characteristics of traffic in units of flow (a collection of data packets with the same source IP address, destination IP address, source port, destination port, and protocol), providing a data basis for subsequent traffic analysis.

[0013] (1.2) Use the CPU affinity of DPDK to bind several different tasks to different CPUs to complete the operations of acquisition program control, network data flow acquisition, network data packet analysis, and network flow data persistence:

[0014] (1.2.1) Control core: mainly responsible for the control of the acquisition program, statistics of each CPU core in the acquisition program, and control of the start and end of the acquisition program;

[0015] (1.2.2) Collection core: mainly responsible for the collection of data packets, mainly obtaining network traffic data packets from the lock-free ring buffer queue (RX queue) of the network adapter in a polling manner;

[0016] (1.2.3) Analysis core: mainly responsible for data packet analysis, extracting the five-tuple information of the data packet by parsing the protocols of each layer of the data packet, and extracting the spatiotemporal characteristics of the traffic in units of flows;

[0017] (1.2.4) Writing core: Mainly responsible for the persistence of data packets, writing data packets in the form of pcap files, and writing the extracted flow features into the database regularly.

[0018] (1.3) The overall specific process of high-performance traffic collection and storage is as follows:

[0019] (1.3.1) The control core accepts command parameters, initializes the DPDK acquisition environment according to the command parameters, sets the corresponding hardware in the system (such as the CPU core and the memory thread pool), binds different tasks to the corresponding CPU cores, and starts the tasks;

[0020] (1.3.2) The acquisition core obtains network traffic packets from the RX queue of the network adapter in a polling manner, receives packets in batches from the network adapter, and queues them into each corresponding lock-free ring buffer queue;

[0021] (1.3.3) The analysis core takes out data packets in batches from the RX queue, parses the protocol format of each layer of the data packet, extracts the five-tuple information of the data packet, and determines whether the source IP address / destination IP address of the data packet already exists, thereby updating the spatiotemporal characteristics of the flow.

[0022] (1.3.4) The writing core takes out the data packet from the RX queue and writes it into the pcap file, and at the same time periodically persists the device's flow feature table into the database.

[0023] Furthermore, the step (2) specifically includes the following sub-steps:

[0024] (2.1) Unlike high-performance traffic collection and storage, step (2) aims to provide effective label data set support for the encrypted traffic service classification model without focusing on traffic collection performance. Due to the construction requirements of the service classification model, the present invention includes the process of automatic sample collection, manual intervention sample label construction, sample purification and feature extraction.

[0025] Furthermore, the step (2.1) specifically includes the following sub-steps:

[0026] (2.1.1) A network traffic collection and sample construction method based on the "PID (Process IDentifier)-port" pair is proposed. By automatically extracting the mapping relationship between the process PID and the network port within the time window, the correspondence between the process and the network traffic is achieved. In the network system scenario, the traffic of each process is isolated and collected and preliminary samples are constructed.

[0027] (2.1.2) In view of the possible one-to-many mapping relationship between business applications and processes, the traffic sample association of multi-process network services is realized through screening. Based on the process traffic samples, the traffic between multiple processes of a single application is online associated to obtain complete pure business traffic samples, and a hash mapping with one-to-one correspondence between sample traffic and sample labels is established.

[0028] (2.1.3) Perform out-of-order recovery and retransmission deduplication, filtering and purification, and protocol data unit feature extraction operations on the original collected encrypted traffic. At the same time, selectively remove weakly related traffic data such as ACK messages, handshake messages, and FIN messages, filter specific encrypted traffic with information such as fixed ports and fixed protocol header formats, retain highly sensitive features of business classification, create tables based on key values ​​such as target IP addresses, and store the data features of the collected traffic in the database.

[0029] (2.2) In order to adapt to the accuracy and efficiency of encrypted traffic anomaly detection in real scenarios, the present invention designs a multi-level business classification scheme based on heterogeneous features, fully utilizing the metadata features, statistical features, sequence features and other heterogeneous features of the collected sample traffic, and adopts traditional ensemble learning, deep learning and other classification models to provide reliable traffic pattern support for the implementation of step (3). The output model will be connected to the database in step (1), and the collected encrypted traffic will be classified and predicted based on fixed time intervals, and the prediction results will be persisted in the database for subsequent anomaly detection queries.

[0030] Furthermore, the step (2.2) specifically includes the following sub-steps:

[0031] (2.2.1) In order to improve the efficiency of business classification, the present invention constructs a lightweight business classification model based on the statistical characteristics of sample flows, calculates macro-statistical characteristics based on the traffic metadata characteristics of the business classification database, extracts valuable features respectively, uniformly constructs flow feature vectors, and reduces and integrates them using the principal component analysis method. The mined highly expressive flow feature vectors are used as input data, and representative integrated classification models such as random forest, Adaboost, and Xgboost are selected. The optimal one is used as the lightweight business classification model based on the training results, and finally efficient and fast classification with acceptable accuracy is achieved.

[0032] (2.2.2) Aiming at the high accuracy of business classification, the present invention proposes to use the multi-protocol data unit length sequence as a feature, and construct a length-sensitive deep learning model based on the feature extraction method of the sliding supersequence and N-gram model. The fine-grained sequence features with stronger representation ability are used as input, and the flow length sequence features are read from the business classification database as a sample data set, so as to establish well-known sequence classification models such as LSTM, Fs-Net, and CapNet, complete the training and testing of the model in a deep learning manner, and select the best one as the high-precision business classification model according to the training results.

[0033] (2.2.3) In view of the comprehensive requirements of business classification indicators (ensuring high efficiency and high precision), the present invention comprehensively evaluates the advantages and disadvantages of lightweight classification models and high-precision classification models, and proposes a multi-model joint precision classification method to greatly improve the efficiency and reliability of classification results, and effectively reduce the risk and maintenance cost of later abnormal detection. The present invention successively applies lightweight classification models and high-precision classification models in a series form, manually sets a threshold in the results of the lightweight classification model, regards the high-confidence results as the final prediction value, and re-inputs the data with confidence lower than the threshold into the high-precision classification model for secondary classification. The appropriate threshold is finally selected through manual debugging to achieve a balance between classification efficiency and accuracy.

[0034] Furthermore, the step (3) specifically includes the following sub-steps:

[0035] (3.1) Based on the device flow table in the system database, the flow is divided into a given time slice size and a flow profile of the device is constructed. The specific process is as follows:

[0036] (3.1.1) The target IP address is used as the basis for space division. The target IP address is the specified host IP (the target IP address is not necessarily the destination IP address in the traffic quintuple). For example, all host IPs belonging to a certain subnet segment in the intranet. If the source IP or destination IP of the network flow is the same as the target IP, the flow will be divided under this target IP.

[0037] (3.1.2) After high-performance traffic collection, the time dimension is arranged according to the start timestamp of the flow, and the time period division of the network flow is completed under the condition of setting the time slice size. In this way, the network traffic division in the time and space dimensions is realized.

[0038] (3.1.3) Use IP as the table name to build traffic information tables for different devices. The table attributes include underlying features such as flow timestamp, flow five-tuple information, flow length sequence, and multiple flow statistical features. It also includes labels predicted by the business model for each flow. Therefore, all table entry information in each table constitutes the traffic profile of the corresponding IP device.

[0039] (3.2) Based on the device-level traffic profile information, a variety of anomaly evaluation strategies are proposed and summarized into an anomaly detection mechanism to build an anomaly detection model with multiple strategies. On the basis of making full use of traffic metadata information, statistical features, sequence features and business category information, reliable and complete device-level anomaly detection is achieved, and the detection mechanism is applied to system implementation, and early warning information is issued in a timely manner when a specified anomaly is detected. The specific process is as follows:

[0040] (3.2.1) Traffic profiling-based anomalies are different from intrusion traffic and malicious traffic. They mainly explore the distribution patterns of network traffic in time and space dimensions, and then warn of possible anomalies of a device based on prior knowledge. The following four potential anomalies are proposed:

[0041] Abnormal device traffic scale: When a device fails, communication latency is high, or it is attacked by DDos, the network traffic scale of the device will be significantly different from its prior rules, which is considered to be abnormal device traffic scale.

[0042] Business label anomaly: By dynamically adjusting the confidence threshold, the flow with the highest confidence less than the confidence threshold is regarded as unknown business, and the "unknown business" label is added to it, and it is classified as business label anomaly.

[0043] Abnormal business traffic distribution: Based on the predicted business label attributes in the device-level traffic profile, the business category distribution of each device in the designated time slice is analyzed. If the difference between the business category distribution and the prior distribution of the device exceeds the indicator range, it is considered that the business traffic distribution is abnormal.

[0044] Business traffic time anomaly: Based on the predicted business label attributes in the device-level traffic profile, analyze the distribution of each business category of the device in the time dimension. If the difference between it and the prior distribution of the device exceeds the indicator range, it is considered as a business traffic distribution anomaly.

[0045] (3.3) Based on the above potential anomalies, develop anomaly detection strategies for device traffic scale anomalies, service label anomalies, service traffic distribution anomalies, service traffic time anomalies, etc., and integrate each strategy to generate an asynchronous anomaly detection mechanism. In the system application, each strategy is executed and evaluated in parallel to improve the response speed of the anomaly detection process.

[0046] (3.3.1) For abnormal device traffic scale: According to the IP device-level traffic profile table, set the time slice to build a sliding window, select data items through the sliding window, and analyze the selected data items with data analysis methods. Use the previous window information as prior information, and use indicators such as Euclidean distance and Manhattan distance to measure the difference between the current information and the prior information. Under the condition of setting a threshold, if the difference between the previous and the next exceeds the threshold, it is considered that an abnormal device traffic scale has occurred, and the abnormal information is prompted.

[0047] (3.3.2) For business tag anomalies: Based on the IP device-level traffic profile table, query the confidence of the business tag attributes. Under the condition of the set threshold, if a business tag with a confidence level lower than the threshold appears, it is considered a business tag anomaly and the anomaly information is prompted.

[0048] (3.3.3) For abnormal business traffic distribution: Based on the IP device-level traffic profile table, count the number of each business label in the current window, use the previous window information as prior information, and use indicators such as Mahalanobis distance and Euclidean distance to measure the difference between the current information and the prior information. Under the condition of a given threshold, if the difference between the before and after exceeds the threshold, it is considered that an abnormal business traffic distribution has occurred, and the system will prompt the abnormal information.

[0049] (3.3.4) For service traffic time anomalies: According to the IP device-level traffic profile table, set a larger time slice, count the distribution of each service label in the time slice, summarize the service traffic pattern with data analysis methods, and use the front window information as a priori information for comparison. If the traffic pattern of a certain service label (difference with the prior information exceeds the acceptable range, it is considered that a service traffic time anomaly has occurred, and the abnormal information is prompted.

[0050] The constructed device portrait and anomaly detection model inevitably faces the problem of concept drift and cannot continuously achieve accurate business classification and anomaly detection. Therefore, the present invention proposes a life cycle maintenance method for device portrait and detection model, which is guided by the idea of ​​self-evolution, continuously collects the latest business traffic samples and sets an available time period to update traffic samples and device traffic portraits, ultimately ensuring the continuous effectiveness and availability of the traffic anomaly detection function.

[0051] The device-level traffic profile is the data source of the anomaly detection mechanism of the present invention, so the periodic update of the device traffic profile is an important part of the present invention. Therefore, the present invention proposes to discard the early device traffic profile in a fixed time period and continuously supplement the newly collected profile information so that the profile data in the database is more in line with the actual status of the current network. Timely updating and maintaining the device-level traffic profile is conducive to improving the overall efficiency of the system and the accuracy of model reasoning.

[0052] Since the anomaly detection mechanism of the present invention is mainly based on device-level traffic profiling, and the business label therein is an important basis for anomaly detection, the present invention proposes to collect business traffic samples in a fixed time period through automated online encrypted traffic collection and sample construction, expand and update the original business data set, and use the idea of ​​incremental learning to achieve continuous learning of the business classification model while maintaining the learned features of the business classification model, thereby providing continuous and reliable advanced traffic features for business-oriented encrypted traffic anomaly detection.

[0053] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0054] (1) High-speed traffic collection provides real data for anomaly detection and is applicable to high-speed network environments. High-performance traffic collection is mainly based on DPDK for high-speed traffic collection, which can capture and collect high-speed traffic in network data centers. At the same time, it extracts the spatiotemporal characteristics of traffic in units of flow (a collection of data packets with the same source IP address, sink IP address, source port, sink port, and protocol), and persists the extracted spatiotemporal characteristics in the traffic database, providing a data basis for subsequent traffic analysis.

[0055] (2) Multi-level encrypted traffic service classification In order to meet the specific service classification requirements for encrypted traffic, a two-stage technology of automated online encrypted traffic collection and sample construction, and service classification model construction is proposed to realize the system function. The automated online encrypted traffic collection and sample construction part will sequentially perform automated sample collection and preliminary labeling, sample label construction, sample purification and feature extraction. Multi-level service classification based on heterogeneous features is divided into efficient service classification based on lightweight models, high-precision service classification based on deep learning models, and balanced service classification based on multi-model combination.

[0056] (3) Business-oriented encrypted traffic anomaly detection is based on the realization of "high-performance traffic collection and storage method" and "multi-level encrypted traffic service classification". Business-oriented encrypted traffic anomaly detection is based on high-performance traffic collection and multi-level encrypted traffic service classification. Business-oriented encrypted traffic anomaly detection proposes the construction of device-level traffic portraits, realizes the spatiotemporal granularity division of traffic information and prediction results from the construction level of the system database, and realizes the requirements of this project for the spatiotemporal division of service classification results; proposes anomaly detection for service classification, proposes a variety of abnormal situations and corresponding evaluation strategies based on the device-level traffic in high-speed network scenarios, and builds a traffic anomaly detection model based on this, and systematically implements the anomaly detection function, and timely warns of abnormal situations in online monitoring, meeting the anomaly detection needs of this project for service classification; in response to the concept drift problem in the field of traffic analysis, a life cycle maintenance method for device-level traffic portraits and service classification models is designed based on the self-evolution strategy.

[0057] (4) This patent implements encrypted traffic service classification in multiple demand scenarios and business-oriented encrypted traffic anomaly detection services through high-performance traffic collection and storage, multi-level encrypted traffic service classification, and business-oriented encrypted traffic anomaly detection. It provides support services for the entire process of traffic collection-service classification-anomaly detection, responds to traffic behavior anomalies of service equipment in a timely manner, and ensures the effectiveness and reliability of network and information system operations. At the same time, it uses an incremental self-evolution strategy to dynamically update portraits and models, ensure the longevity of the anomaly detection function, and ensure continuous encrypted traffic supervision in high-speed networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is an encrypted traffic anomaly detection process based on high-performance traffic collection.

[0059] Specific implementation:

[0060] The technical solution provided by the present invention will be described in detail below in conjunction with specific embodiments. It should be understood that the following specific implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0061] The present invention proposes a method for detecting anomalies of encrypted traffic based on high-performance traffic collection, which mainly consists of three parts: high-performance traffic collection and storage, multi-level encrypted traffic service classification, and business-oriented encrypted traffic anomaly detection. Among them, the first part is high-performance traffic collection and storage, which is mainly based on DPDK for high-speed traffic collection, to achieve the capture and collection of high-speed traffic in the network data center, extract features and persist them in the traffic database. The second part is multi-level encrypted traffic service classification, which proposes a two-stage technology of automated online encrypted traffic collection and sample construction, and business classification model construction to achieve specific business classification of encrypted traffic. The third part is business-oriented encrypted traffic anomaly detection, which proposes the construction of device-level traffic portraits based on the first and second parts, and builds a traffic anomaly detection model based on various abnormal situations and corresponding evaluation strategies of device-level traffic in real network scenarios to achieve real-time anomaly warning. At the same time, in view of the concept drift problem in the field of traffic analysis, a life cycle maintenance method of device-level traffic portraits and business classification models is designed based on the self-evolution strategy.

[0062] Specifically, the present invention has the following steps:

[0063] (1) Based on DPDK, high-performance traffic collection and storage are performed to capture and collect high-speed traffic in network data centers. At the same time, the spatiotemporal characteristics of traffic are extracted based on flows (a collection of data packets with the same source IP address, sink IP address, source port, sink port, and protocol). The extracted spatiotemporal characteristics are persisted in the traffic database to provide a data basis for subsequent traffic analysis.

[0064] The specific process of this step is as follows:

[0065] (1.1) Through the CPU affinity of DPDK, the control plane threads and each data plane thread are bound to different CPU cores, avoiding the performance consumption of repeated scheduling between CPUs and giving full play to the performance of multi-core CPUs. Therefore, in the design of traffic collection, according to the actual needs of traffic collection and the characteristics of the DPDK framework, several different tasks are bound to different CPUs to complete the operations of collection program control, network data flow collection, network data packet analysis, and network flow data persistence;

[0066] (1.2) Use the CPU affinity of DPDK to bind several different tasks to different CPUs to complete the operations of acquisition program control, network data flow acquisition, network data packet analysis, and network flow data persistence.

[0067] (1.2.1) Control core: mainly responsible for the control of the collection program, statistics of each CPU core in the collection program (the number of collected packets, number of bytes, packet loss rate, etc.), and control of the start and end of the collection program;

[0068] (1.2.2) Collection core: mainly responsible for data packet collection, in the form of polling, this task is mainly to obtain network traffic data packets from the RX queue of the network adapter, the RX queue is the data packet receiving queue. After obtaining the network traffic data packet data, add the corresponding metadata information before the traffic data, and use the rte_ring_enqueue_burst() function to queue the processed traffic data into each corresponding RX queue.

[0069] (1.2.3) Analysis core: mainly responsible for data packet analysis, extracting the five-tuple information of the data packet (source IP address, sink IP address, source port, sink port, protocol) by parsing the protocols of each layer of the data packet, and extracting the spatiotemporal characteristics of the traffic in units of flow;

[0070] (1.2.4) Writing core: Mainly responsible for the persistence of data packets, writing data packets in the form of pcap files, and writing the extracted flow features into the database regularly.

[0071] (1.3) The overall specific process of high-performance traffic collection and storage is as follows:

[0072] (1.3.1) First, the control core accepts the command parameters and initializes the DPDK acquisition environment EAL according to the command parameters. During the initialization of the EAL layer, the lcore will be initialized. In the DPDK function, the rte_eal_cpu_init() function will be called to obtain the device data in the corresponding file of the device information in the system, thereby obtaining information such as which CPU cores are available in the current system, and setting the parameters through the eal_parse_args() function. At the same time, the control core will create a memory pool (rte_mempool), configure and start the network card parameters (determine whether the network card device is valid, configure and start the RX queue, turn on the promiscuous mode, etc.), bind the callback functions of different tasks to their respective CPU cores, and start the tasks;

[0073] (1.3.2) Secondly, the acquisition core obtains network traffic packets from the RX queue of the network adapter in a polling manner, receives packets in batches from the network adapter, and queues them into each corresponding lock-free ring buffer receiving queue (rx_ring);

[0074] (1.3.3) Then, the analysis core takes out data packets in batches from rx_ring, parses the protocol format of each layer of the data packet, extracts the five-tuple information of the data packet, and determines whether the source IP address / destination IP address of the data packet already exists: if not, a new device IP feature flow table is created and the spatiotemporal features of the flow are added; if already exists, check whether the flow of the data packet already exists in the device IP feature flow table: if not, the spatiotemporal features of the flow are added; if existing, the spatiotemporal features of the flow are updated.

[0075] (1.3.4) The writing core takes out the data packet from rx_ring and writes it into the pcap file, and at the same time periodically persists the device's flow feature table into the database.

[0076] (2) Aiming at the specific business classification of encrypted traffic in large-scale network scenarios, a two-stage technology of automated online encrypted traffic collection and sample construction, and business classification model construction is proposed to realize encrypted traffic business classification. The present invention successively realizes automated online encrypted traffic collection and sample construction, and multi-level business classification based on heterogeneous features, wherein the automated online encrypted traffic collection and sample construction part will sequentially perform automated sample collection and preliminary marking, sample label construction, sample purification and feature extraction; the multi-level business classification based on heterogeneous features is divided into efficient business classification based on lightweight models, high-precision business classification based on deep learning models, and balanced business classification based on multi-model combination.

[0077] The specific process of this step is as follows:

[0078] (2.1) Different from the high-performance traffic collection and storage in the first part, the present invention aims to provide effective label data set support for the encrypted traffic service classification model without paying attention to the traffic collection performance. Therefore, the steps of automatic sample collection, manual intervention sample label construction, sample purification and feature extraction are designed.

[0079] (2.1.1) In order to solve the problem of background traffic interference in traffic label sample collection, a network traffic collection and sample construction method based on "PID (Process IDentifier)-port" pairs is implemented. This method breaks through the limitation that traditional methods require a pure application running environment. By automatically extracting the mapping relationship between the process PID and the network port within the time window, the correspondence between the process and the network traffic is realized. In the real network system scenario, the traffic of each process is isolated and collected and preliminary samples are constructed.

[0080] (2.1.2) In view of the possible one-to-many mapping relationship between business applications and processes, the traffic sample association of multi-process network services is realized through later manual screening. Based on the process traffic samples obtained in the early stage, the traffic between multiple processes of a single application is online associated to obtain complete pure business traffic samples. Finally, the collection of labeled traffic samples guided by business categories is realized, and the labeling of sample business labels is further realized with the help of controllable devices, and a hash mapping of sample traffic and sample labels is established with one-to-one correspondence.

[0081] (2.1.3) Perform out-of-order recovery, retransmission deduplication, filtering and purification, and protocol data unit feature extraction operations on the original collected encrypted traffic. For example, in order to address the errors between the spliced ​​traffic and the original network data of the application caused by TCP out-of-order and message retransmission, adopt the solution of TCP out-of-order reordering and retransmission message deduplication on the original traffic to better restore the business data characteristics. At the same time, selectively clear weakly related traffic data such as ACK messages, handshake messages, and FIN messages, filter specific encrypted traffic with information such as fixed ports and fixed protocol header formats, and retain highly sensitive characteristics of business classification. Finally, store the traffic data characteristics generated by different devices in blocks, create tables based on key values ​​such as target IP addresses, and store the traffic data characteristics collected by different devices in the business classification database, providing a data basis for establishing a highly robust business classification model in the next stage.

[0082] (2.2) Considering the requirements of accurate and efficient encrypted traffic anomaly detection in real scenarios, the present invention designs a multi-level service classification technology based on heterogeneous features, making full use of the metadata features, statistical features, sequence features and other heterogeneous features of the collected sample traffic, and adopting traditional ensemble learning, deep learning and other classification models. On the basis of meeting the needs of multi-level service classification, it also provides reliable traffic pattern support for the implementation of the third part of anomaly detection. At the same time, the multi-level service classification model will be connected to the first part of the database, and the collected encrypted traffic will be classified and predicted based on fixed time intervals, and the prediction results will be persisted in the database for subsequent anomaly detection queries.

[0083] The specific process of this step is as follows:

[0084] (2.2.1) Aiming at the high efficiency requirement of business classification, the present invention constructs a lightweight business classification model based on sample flow statistical features, extracts valuable features such as adjacent message intervals, number of bytes transmitted per second, flow duration, flow active time, and number of uplink and downlink messages based on the traffic metadata features of the business classification database and calculates macro-statistical features, uniformly constructs flow feature vectors, and reduces and integrates them using the principal component analysis method, using the mined highly expressive flow feature vectors as input data. Select representative integrated classification models such as random forest, Adaboost, and Xgboost, and select them as lightweight business classification models based on the training results, so as to achieve efficient and rapid classification with acceptable accuracy and improve the overall efficiency of the entire process of each stage of anomaly detection.

[0085] (2.2.2) In response to the high-precision requirements for business classification, the present invention proposes to use the multi-protocol data unit length sequence as a feature, and based on the feature extraction method of the sliding supersequence and N-gram model, construct a length-sensitive deep learning model, take the fine-grained sequence features with stronger representation capabilities as input, read the flow length sequence features from the business classification database as a sample data set, establish well-known sequence classification models such as LSTM, Fs-Net, and CapNet, complete the training and testing of the model in a deep learning manner, and select the best model as a high-precision business classification model based on the training results.

[0086] (2.2.3) In view of the comprehensive requirements of business classification indicators, the present invention comprehensively evaluates the advantages and disadvantages of lightweight classification models and high-precision classification models, and proposes a multi-model joint precision classification method to greatly improve the efficiency and reliability of classification results, and effectively reduce the risk and maintenance cost of later abnormal detection. The present invention successively applies lightweight classification models and high-precision classification models in a series form, sets a threshold in the results of the lightweight classification model, regards the high-confidence results as the final prediction value, and inputs the data with confidence lower than the threshold into the high-precision classification model for secondary classification. By debugging and selecting a suitable threshold, a balance between classification efficiency and accuracy is achieved.

[0087] (3) Business-oriented encrypted traffic anomaly detection Based on high-performance traffic collection and multi-level encrypted traffic service classification, the construction of device-level traffic portraits is proposed, and the spatiotemporal granularity division of traffic information and prediction results is realized from the construction level of the system database. Based on the various abnormal situations and corresponding evaluation strategies of device-level traffic in real network scenarios, a traffic anomaly detection model is constructed to realize online anomaly detection. In addition, in response to the concept drift problem in the field of traffic analysis, the present invention designs a life cycle maintenance method for device-level traffic portraits and service classification models based on a self-evolution strategy, which fully meets all aspects of business-oriented traffic anomaly detection.

[0088] The specific process of this step is as follows:

[0089] (3.1) The traffic data and business classification results are divided in the time dimension and space dimension, and on this basis, a device-level traffic profile is constructed to fully describe the traffic situation passing through the target IP device, and then reflect the operating status of the device from the traffic perspective, thereby providing reliable evaluation information for the next stage of anomaly detection tasks.

[0090] (3.1.1) First, the target IP address is used as the basis for space division. The target IP address is the designated host IP of interest (the target IP address is not necessarily the destination IP address in the traffic quintuple), such as all host IPs belonging to a certain subnet segment in the intranet. If the source IP or destination IP of the network flow is the same as the target IP, the flow will be divided under this target IP.

[0091] (3.1.2) Secondly, after high-performance traffic collection, the time dimension is arranged according to the start timestamp of the flow, and the network traffic division in the time and space dimensions is completed under the condition of manually setting the time slice size, and the device-level traffic portrait is constructed based on the IP division. Specifically, the present invention constructs the traffic information table of different devices with IP as the table name. The attributes of the table include underlying features such as flow timestamp, flow five-tuple information, flow length sequence, and various flow statistical features, and also include labels predicted by the business model for each flow. Therefore, all the table item information of each table constitutes the traffic portrait of the corresponding IP device.

[0092] (3.2) Based on the device-level traffic profile information, the present invention proposes a variety of anomaly assessment strategies, which are summarized into an anomaly detection mechanism, and constructs a multi-strategy integrated anomaly detection model. Ultimately, based on the full use of traffic metadata information, statistical features, sequence features, and business category information, reliable and complete device-level anomaly detection is achieved, and early warning information is issued in a timely manner when a specified anomaly is detected.

[0093] (3.2.1) Traffic profiling-based anomalies are different from intrusion traffic and malicious traffic. They mainly explore the distribution patterns of network traffic in the time and space dimensions, and then warn of possible anomalies of a device based on prior knowledge. Since the device-level traffic profiling records the traffic information of different IPs in the time dimension, the anomalies in coarse-grained time periods (a longer period of time) and fine-grained time periods (a shorter period of time) can be dynamically analyzed based on this information. The present invention sets the following potential anomalies:

[0094] Abnormal device traffic scale: The device-level traffic profile records the detailed traffic information of each IP device in chronological order. The total amount of network traffic generated by each device in a fixed time period (such as one day) should remain relatively stable. The traffic information of each IP in different time slices is analyzed by manually setting time slices, and the differences in time dimensions are compared. Therefore, when there is a device failure, high communication latency, or DDos attack, the network traffic scale of the device will be significantly different from its prior rules, which is considered to be abnormal device traffic scale.

[0095] Business label abnormality: Each encrypted flow in the device-level traffic profile will contain a predicted business label after being predicted by the "Multi-level Encrypted Traffic Business Classification" module. During the prediction process of the multi-classification model, the confidence of the flow in each business category will be generated, and the model will use the category with the highest confidence as the prediction result. Therefore, by manually and dynamically adjusting the confidence threshold, the flow with the highest confidence less than the confidence threshold is regarded as an unknown business, and an "unknown business" label is added to it. Based on the prior knowledge of different devices, it can be judged whether the business label abnormality appears in the device profile of the current time slice.

[0096] Abnormal business traffic distribution: Based on the predicted business label attributes in the device-level traffic profile, the business category distribution of each device in the designated time slice can be analyzed. When the evaluation index is set, if the difference between it and the prior distribution of the device exceeds the index range, it is considered that the business traffic distribution is abnormal.

[0097] Business traffic time anomaly: Based on the predicted business label attributes in the device-level traffic profile, the distribution of each business category of the device in the time dimension can be analyzed. When the evaluation index is set, if the difference between it and the prior distribution of the device exceeds the index range, it is considered that the business traffic distribution is abnormal.

[0098] (3.2.2) Based on the above potential anomalies, formulate anomaly detection strategies for device traffic scale anomalies, service label anomalies, service traffic distribution anomalies, service traffic time anomalies, etc., and integrate each strategy to generate an asynchronous anomaly detection mechanism, execute each strategy in parallel for evaluation, and improve the response speed of the anomaly detection process. The specific anomaly detection strategy of the present invention is as follows:

[0099] For abnormal device traffic scale: Based on the system database, query the IP device-level traffic portrait table respectively, set the multi-level time slice scale (such as 1 hour, 12 hours, 24 hours), and construct the sliding window of the corresponding scale respectively. Sort the table items in order of timestamp, and select data items with a sliding window. Use data analysis methods to analyze the total number of bytes, total number of messages and other information of the selected network flow, use the pre-window information as prior information, and measure the difference between the current information and the prior information with indicators such as Euclidean distance and Manhattan distance. Under the condition of setting the threshold, if the difference between the previous and the next exceeds the threshold, it is considered that the device traffic scale is abnormal, and the system prompts the abnormal information.

[0100] For business tag anomalies: Based on the system database, query the IP device-level traffic profile table separately, and query the confidence of the business tag attributes. Under the condition of the set threshold, if a business tag with a confidence level lower than the threshold appears, it is considered a business tag anomaly and the system prompts the anomaly information.

[0101] For abnormal business traffic distribution: Based on the system database, query the IP device-level traffic portrait table separately, count the number of each business label in the current window, use the previous window information as prior information, and measure the difference between the current information and the prior information with indicators such as Mahalanobis distance and Euclidean distance. Under the condition of a given threshold, if the difference between the before and after exceeds the threshold, it is considered that the business traffic distribution is abnormal, and the system prompts the abnormal information.

[0102] For business traffic time anomalies: Based on the system database, query the IP device-level traffic portrait table separately, set a larger time slice scale (such as 24 hours), and count the distribution of each business label in the time slice. Use data analysis methods to summarize the business traffic pattern, and use the preceding window information as prior information for comparison. If the traffic pattern of a certain business label (such as business traffic peak period, idle period) differs from the prior information by more than an acceptable range, it is considered that a business traffic time anomaly has occurred, and the system prompts the abnormal information.

[0103] (3.3) Due to the temporal differences in the time and space dimensions of real network traffic, the device portraits and anomaly detection models constructed in the early stage inevitably face the problem of concept drift, which in turn leads to a short life cycle of the classification model and anomaly detection mechanism, and cannot continuously achieve accurate business classification and anomaly detection. The present invention proposes a life cycle maintenance method for device portraits and detection models, which is guided by the idea of ​​self-evolution, continuously collects and persists the latest business traffic samples, manually sets the available time period of the portraits and models, and eliminates old traffic samples and device traffic portraits based on this to maintain the periodic update of data and portraits. At the same time, the model is automatically iterated based on the time period, and finally the continuous effectiveness and availability of the traffic anomaly detection function are guaranteed. Therefore, the present invention significantly reduces the risk of model unavailability caused by the concept drift problem by automatically updating the model, thereby minimizing the product maintenance cost, reflecting the characteristics of the present invention of high quality, low risk and easy maintenance.

[0104] (3.3.1) The device-level traffic profile is the data source of the anomaly detection mechanism of the present invention, so the periodic update of the device traffic profile is an important part of the present invention. In a real network scenario, as time goes by, the device traffic profile in the system database will expand rapidly. On the one hand, this will take up a lot of storage space, resulting in a decrease in query efficiency, which in turn affects the efficiency of the whole process reasoning. On the other hand, due to the Markov property of traffic data, the device traffic profile in the present invention is often only associated with the first N windows (N can be dynamically adjusted according to demand). Therefore, it is proposed to discard the early device traffic profile in a fixed time period and continuously supplement the newly collected profile information to make the profile data in the database more in line with the current network reality. Real state.

[0105] (3.3.2) Since the anomaly detection mechanism of the present invention is mainly based on device-level traffic profiling, and the business label therein is an important basis for anomaly detection, the accuracy of the data attribute largely determines the anomaly detection effect, and thus the self-evolution of the business classification data and model is an important part of maintaining the cycle of this project. The present invention proposes to collect business traffic samples through automated online encrypted traffic collection and sample construction in a fixed time period, expand and update the original business data set, and use the idea of ​​incremental learning to increase the model's feature recognition of new sample data while maintaining the learned features of the business classification model, thereby realizing continuous learning of the business classification model and providing continuous and reliable advanced traffic features for business-oriented encrypted traffic anomaly detection.

[0106] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. An encrypted traffic anomaly detection method based on high-performance traffic acquisition, characterized in that The steps include: (1) High-performance traffic acquisition based on DPDK realizes the capture and acquisition of high-speed traffic, and at the same time extracts traffic characteristics in units of flow and persists them into the traffic database; (2) Based on the business classification of encrypted traffic, it proposes two-stage technology of automated online encrypted traffic collection, sample construction, and business classification model construction to realize specific business classification for encrypted traffic; (3) Business-oriented encrypted traffic anomaly detection builds a traffic anomaly detection model based on various abnormal situations of device-level traffic in a real network environment and corresponding evaluation strategies to provide timely warnings of abnormal situations.

2. The encrypted traffic anomaly detection method based on high-performance traffic acquisition according to claim 1, characterized in that, The step (1) specifically includes the following sub-steps: (1.1) Use the CPU control core to accept command parameters, initialize the DPDK acquisition environment according to the command parameters, and bind different tasks to their respective CPU cores and start tasks; (1.2) Obtain network traffic packets from the network card in the form of polling through the CPU acquisition core and send them to each corresponding lock-free ring buffer queue (RX queue); (1.3) Use the CPU analysis core to batch-take data packets from the RX queue, parse the protocol formats of each layer of the data packet, extract the five-tuple information and corresponding features of the data packet, and update the corresponding features of the flow based on whether the flow already exists; (1.4) Use the CPU write core to retrieve the packet from the RX queue and write it to the pcap file, and at the same time persist the device's stream feature table to the database regularly.

3. The method for detecting anomaly of encrypted traffic based on high-performance traffic acquisition according to claim 2, characterized in that, The specific requirements of step (2) are: (2.1) The automated online encrypted traffic acquisition and sample construction part will perform automated sample collection and preliminary marking, sample label construction, sample purification and feature extraction in sequence; (2.2) Automated sample collection and preliminary labeling By automatically extracting the mapping relationship between the process PID and the network port within the time window, the correspondence between the process and the network traffic is achieved, and the isolated traffic collection and preliminary sample construction of each process are completed; (2.3) Business category-oriented label traffic sample collection: Based on the process traffic samples, the traffic between multiple processes of a single application service is online associated to achieve the labeling of sample service labels, and a hash mapping is established with one-to-one correspondence between sample traffic and sample labels; (2.4) Sample purification and feature extraction realizes out-of-order recovery, retransmission deduplication, filtering and purification, and protocol feature extraction operations, while selectively removing weakly related traffic data, filtering specific encrypted traffic with fixed information, retaining highly sensitive features of business classification, and creating tables by target IP address key value classification to store sample traffic data features collected by the corresponding device; (2.5) Multi-level service classification based on heterogeneous features is divided into efficient service classification based on lightweight models, high-precision service classification based on deep learning models, and balanced service classification based on multi-model combination; (2.6) A lightweight service classification model is constructed based on the statistical characteristics of sample flows. The traffic metadata characteristics of the service classification database are used to calculate the macro-statistical characteristics, and the principal component analysis method is used to reduce and integrate them. The ensemble learning classification model is used to select the best results as the lightweight service classification results, achieving efficient and fast classification with acceptable accuracy. (2.7) Taking the length sequence of multi-protocol data units as features, based on the feature extraction method of sliding supersequence and N-gram model, the length-sensitive deep learning model is constructed with the well-known sequence classification models of LSTM, Fs-Net, and CapNet. The model is trained and tested in a deep learning manner, and the best model is selected as a high-precision business classification model based on the training results; (2.8) Comprehensively evaluate the advantages and disadvantages of lightweight classification models and high-precision classification models, and propose a multi-model joint precision classification method to greatly improve the efficiency and reliability of classification results, and effectively reduce the risk and maintenance cost of subsequent anomaly detection. The solution applies the lightweight classification model and the high-precision classification model in series, manually sets a threshold in the results of the lightweight classification model, regards the high-confidence results as the final prediction value, and inputs the data with confidence lower than the threshold into the high-precision classification model for secondary classification.

4. The encrypted traffic anomaly detection method based on high-performance traffic acquisition according to claim 3, characterized in that, The step (3) specifically comprises the following steps: (3.1) The traffic data and service classification results are divided in the time dimension and space dimension, and on this basis, a device-level traffic profile is constructed to fully describe the traffic situation passing through the target IP device, and then reflect the operating status of the device from the traffic perspective, thereby providing reliable evaluation information for the next stage of anomaly detection tasks; (3.2) Based on the device-level traffic profile information, a variety of anomaly assessment strategies are proposed and summarized into an anomaly detection mechanism. A multi-strategy integrated anomaly detection model is constructed. On the basis of making full use of traffic metadata information, statistical features, sequence features, and business category information, reliable and complete device-level anomaly detection is achieved, and warning information is issued in a timely manner when a specified anomaly is detected; (3.3) Through the life cycle maintenance method of device profiling and detection models, guided by the idea of ​​self-evolution, the latest business traffic samples are continuously collected and persisted, and the available time period of the profiling and model is manually set. Based on this, old traffic samples and device traffic profiling are eliminated to maintain the periodic update of data and profiling. At the same time, the model is automatically iterated based on the time period to ensure the continued effectiveness and availability of the traffic anomaly detection function.

5. The encrypted traffic anomaly detection method based on high-performance traffic acquisition according to claim 2, characterized in that, Step (1.2) is as follows: (1.2.1) Control core: It is mainly responsible for the control of the acquisition program, statistics of the statistical information of each CPU core in the acquisition program, and at the same time control the start and termination of the acquisition program; (1.2.2) Acquisition core: It is mainly responsible for the acquisition of data packets, mainly to obtain network traffic packets from the lockless ring buffer queue (RX queue) of the network adapter through polling; (1.2.3) Analysis core: mainly responsible for the analysis of data packets, extract the five-tuple information of data packets by analyzing the protocols of each layer of the data packet, and extract the spatiotemporal characteristics of traffic in units of streams; (1.2.4) Write kernel: It is mainly responsible for the persistence of data packets, writes the data packets in the form of pcap files, and writes the extracted stream features to the database regularly.

6. The encrypted traffic anomaly detection method based on high-performance traffic acquisition according to claim 5, characterized in that, Steps (1.3) The overall specific process of high-performance traffic acquisition and storage is as follows: (1.3.1) Control the core to accept command parameters, initialize the DPDK acquisition environment according to the command parameters, and set the corresponding hardware in the system. At the same time, bind different tasks to the corresponding CPU core and start tasks; (1.3.2) The acquisition core obtains network traffic data packets from the RX queue of the network adapter in the form of polling, receives data packets in batches from the network card, and joins the queue to each corresponding lock-free ring buffer queue; (1.3.3) The analysis core takes out data packets in batches from the RX queue, parses the protocol formats of each layer of the data packets, extracts the five-tuple information of the data packets, and determines whether the source IP address / destination IP address of the data packets already exists, thereby updating the spatiotemporal characteristics of the flow. (1.3.4) The write core takes out the data packet from the RX queue and writes it to the pcap file, and at the same time persists the device's stream feature table to the database regularly.

7. The encrypted traffic anomaly detection method based on high-performance traffic acquisition according to claim 3, characterized in that, Step (2.1) includes the processes of automated sample collection, manual intervention in sample label construction, sample purification and feature extraction, which specifically includes the following sub-steps: (2.1.1) A network traffic collection and sample construction method based on the "PID (Process IDentifier)-port" pair. By automatically extracting the mapping relationship between the process PID and the network port within the time window, the correspondence between the process and the network traffic is achieved. In the real end system scenario, the traffic of each process is isolated and the preliminary sample is constructed. (2.1.2) In view of the one-to-many mapping relationship between business applications and processes, the traffic sample association of multi-process network services is realized through screening. On the basis of obtaining process traffic samples, the traffic between multiple processes of a single application is online associated to obtain complete pure business traffic samples, and a hash mapping with one-to-one correspondence between sample traffic and sample labels is established. (2.1.3) Perform out-of-order recovery and retransmission deduplication, filtering and purification, and protocol data unit feature extraction operations on the original collected encrypted traffic. At the same time, selectively clear the weakly related traffic data of ACK messages, handshake messages, and FIN messages, filter specific encrypted traffic with fixed ports and fixed protocol header format information, retain highly sensitive features of business classification, create tables by target IP address key value classification, and store the data features of the collected traffic in the database.

8. The encrypted traffic anomaly detection method based on high-performance traffic acquisition according to claim 7, characterized in that, The step (2.2) specifically includes the following sub-steps: (2.2.1) Aiming at the high efficiency of business classification, a lightweight business classification model is constructed based on the statistical characteristics of sample flows. Based on the traffic metadata characteristics of the business classification database and the calculation of macro-statistical characteristics, valuable features are extracted respectively, and the flow feature vector is uniformly constructed. The principal component analysis method is used to reduce and integrate the mined highly expressive flow feature vectors as input data, and representative ensemble classification models such as random forest, Adaboost, and Xgboost are selected. The best one is selected as the lightweight business classification model according to the training results, and finally efficient and fast classification with acceptable accuracy is achieved. (2.2.2) Aiming at the high accuracy of business classification, we use the length sequence of multi-protocol data units as features, and build a length-sensitive deep learning model based on the feature extraction method of sliding supersequence and N-gram model. We use the fine-grained sequence features with stronger representation ability as input, and read the flow length sequence features from the business classification database as the sample data set to build the well-known sequence classification models of LSTM, Fs-Net, and CapNet. We complete the training and testing of the model in a deep learning way, and select the best one as the high-precision business classification model based on the training results. (2.2.3) Aiming at the comprehensive requirements of business classification indicators, the advantages and disadvantages of lightweight classification models and high-precision classification models are comprehensively evaluated, and a multi-model joint precision classification method is proposed to greatly improve the efficiency and reliability of classification results, and effectively reduce the risk and maintenance cost of subsequent anomaly detection. The lightweight classification model and the high-precision classification model are applied in series, and the threshold is manually set in the results of the lightweight classification model. The high-confidence results are regarded as the final prediction value, and the data with confidence lower than the threshold is re-entered into the high-precision classification model for secondary classification. The appropriate threshold is finally selected through manual debugging to achieve a balance between classification efficiency and accuracy.

9. The encrypted traffic anomaly detection method based on high-performance traffic acquisition according to claim 4, characterized in that, Step (3.1) Based on the device traffic table of the system database, divide the traffic at a given time slice size and build a traffic portrait of the device. The specific process is as follows: (3.1.1) The target IP address is used as the basis for space division. The target IP address is the specified host IP (the target IP address is not necessarily the sink IP address in the traffic quintuple). For example, for all host IPs belonging to a certain subnet segment in the intranet, if the source IP or sink IP of the network flow is the same as the target IP, the flow is divided into this target IP. (3.1.2) After high-performance traffic collection, the time dimension is arranged according to the start timestamp of the flow, and the time period division of the network flow is completed under the condition of setting the time slice size. In this way, the network traffic division in the time and space dimensions is realized. (3.1.3) Use IP as the table name to build traffic information tables for different devices. The table attributes include flow timestamp, flow five-tuple information, flow length sequence, and multiple underlying features of flow statistics. It also contains the label predicted by the business model for each flow. Therefore, all the table items in each table constitute the traffic profile of the corresponding IP device. (3.2) Based on the device-level traffic profile information, a variety of anomaly judgment strategies are proposed and summarized into an anomaly detection mechanism. A multi-strategy integrated anomaly detection model is constructed. On the basis of making full use of traffic metadata information, statistical features, sequence features and business category information, reliable and complete device-level anomaly detection is achieved. The detection mechanism is applied to system implementation and early warning information is issued in a timely manner when a specified anomaly is detected. The specific process is as follows: (3.2.1) Traffic profiling-based anomalies are different from intrusion traffic and malicious traffic. They mainly explore the distribution patterns of network traffic in time and space dimensions, and then warn of possible anomalies of a device based on prior knowledge. The following four potential anomalies are proposed: Device traffic scale abnormality: When a device fails, communication latency is high, or it is attacked by DDos, the network traffic scale of the device will be significantly different from its prior rules, which is considered to be abnormal. Business label anomaly: By dynamically adjusting the confidence threshold, the flow with the highest confidence less than the confidence threshold is regarded as unknown business, and the "unknown business" label is added to it, and it is classified as business label anomaly. Abnormal business traffic distribution: Based on the predicted business label attributes in the device-level traffic profile, the business category distribution of each device in the designated time slice is analyzed. If the difference between the business category distribution and the prior distribution of the device exceeds the indicator range, the business traffic distribution is considered abnormal. Business traffic time anomaly: Based on the predicted business label attributes in the device-level traffic profile, analyze the distribution of each business category of the device in the time dimension. If the difference between it and the prior distribution of the device exceeds the indicator range, it is considered as a business traffic distribution anomaly.

10. The encrypted traffic anomaly detection method based on high-performance traffic acquisition according to claim 9, characterized in that, Step (3.3) is as follows: (3.3.1) For abnormal device traffic scale: According to the IP device-level traffic profile table, set the time slice to build a sliding window, select data items through the sliding window, and analyze the selected data items with the data analysis method. Use the previous window information as prior information, and use the Euclidean distance and Manhattan distance indicators to measure the difference between the current information and the prior information. Under the condition of setting the threshold, if the difference between the previous and the next exceeds the threshold, it is considered that the device traffic scale is abnormal, and the abnormal information is prompted. (3.3.2) For business tag anomalies: Based on the IP device-level traffic profile table, query the confidence of the business tag attribute. Under the condition of the set threshold, if a business tag with a confidence lower than the threshold appears, it is considered to have a business tag anomaly and prompt the anomaly information. (3.3.3) For abnormal business traffic distribution: According to the IP device-level traffic profile table, the number of each business tag in the current window is counted, the previous window information is used as the prior information, and the Mahalanobis distance and Euclidean distance indicators are used to measure the difference between the current information and the prior information. Under the condition of a given threshold, if the difference between the previous and the next exceeds the threshold, it is considered that the business traffic distribution is abnormal, and the system prompts the abnormal information. (3.3.4) For business traffic time anomalies: Based on the IP device-level traffic profile table, set a larger time slice, count the distribution of each business label in the time slice, summarize the business traffic pattern with data analysis methods, and use the preceding window information as prior information for comparison. If the traffic pattern of a certain business label differs from the prior information by more than an acceptable range, it is considered that a business traffic time anomaly has occurred, and the anomaly information is prompted.

Images