A method for detecting anomalies in encrypted traffic based on high-performance traffic collection

Abstract

The present invention proposes a method for detecting anomalies in encrypted traffic based on high-performance traffic collection, which is as follows: 1) High-performance traffic collection based on DPDK is used to capture and collect high-speed traffic, while extracting traffic features in units of flow and persisting them in a traffic database; 2) Based on the business classification of encrypted traffic, a two-stage technology of automated online encrypted traffic collection and sample construction, and business classification model construction is proposed to achieve specific business classification for encrypted traffic; 3) The construction of device-level traffic portraits is proposed to achieve spatiotemporal granularity division of traffic information prediction results, and a variety of abnormal situations and corresponding evaluation strategies are proposed based on the device-level traffic in the real network environment, and a traffic anomaly detection model is constructed to achieve timely early warning of abnormal situations. The present invention implements support services for the entire process of traffic collection-business classification-anomaly detection, promptly responds to traffic behavior anomalies of each service device, and ensures the effectiveness and reliability of network and information system operations.

Description

Technical Field

[0001] The present invention belongs to the technical field of cyberspace security and relates to a method for detecting anomalies in encrypted traffic based on high-performance traffic collection. Background Art

[0002] With the rapid development of network technology, the scale of the network continues to expand, making the internet an indispensable part of human life. As a carrier of network information, network traffic analysis can be used to understand service types, detect abnormal behavior, detect malicious attacks, and predict link bandwidth. Therefore, network traffic detection has become one of the most important tasks in the network field and a crucial prerequisite for network resource management, cyberspace security, and network service quality. In particular, for cyberspace security, network traffic detection, as an effective network protection method, can detect abnormal network traffic and provide important support for various tasks. It has long attracted widespread attention from scholars.

[0003] However, with the continuous evolution of network encryption and obfuscation technologies, encrypting and obfuscating network traffic has become a common practice in the industry. Encryption and obfuscation alter the traffic fingerprint, statistical characteristics, and traffic distribution of network traffic, making it increasingly difficult to achieve high-speed and accurate network traffic detection and identification. At the same time, various network attack methods on the Internet continue to evolve, and a large number of new network attacks have emerged in cyberspace, significantly impacting cybersecurity. Existing methods for detecting unknown attacks have many shortcomings. Worse still, malicious traffic, aided by encryption and obfuscation technologies, can more easily evade existing network anomaly detection systems, posing a significant threat to network security.

[0004] In order to achieve accurate and efficient encrypted traffic anomaly detection, domestic and foreign researchers usually use a variety of technical means, which can generally be divided into anomaly detection methods based on statistical analysis, detection methods based on feature engineering, detection methods based on traditional machine learning, and detection methods based on deep learning. However, the existing methods are not suitable for high-speed networks and large-scale real network environments. Whether it is traditional machine learning methods or the calculation methods of the popular deep learning methods, they have encountered unprecedented limitations in network security detection, making the existing detection methods unable to meet the real-time and accuracy requirements of attack detection in large-scale high-speed real network environments. Specifically, encrypted traffic anomaly detection has the following problems: (1) The computational cost is high and it is difficult to apply to high-speed network environments; (2) The existing methods are not accurate enough; (3) They focus on traditional protocols and cannot be applied to complex new network protocol scenarios; (4) The model is too complex and is limited by hardware processing capabilities; (5) The analysis method and feature selection rely heavily on manual experience; (6) The methods in the experimental environment ignore the problems and characteristics in the actual network environment. Therefore, the present invention performs high-speed capture and collection of real traffic based on DPDK, and realizes efficient service classification, high-precision service classification and balanced service classification through multi-level encrypted traffic service classification, thereby providing a data basis for subsequent encrypted traffic anomaly detection, building a device-level traffic portrait, and based on various abnormal situations and corresponding evaluation strategies of device-level traffic in real network scenarios, and building a traffic anomaly detection model based on this to realize encrypted traffic anomaly detection. Summary of the Invention

[0005] In view of the various problems of existing encrypted traffic anomaly detection technologies, such as complex calculations, insufficient accuracy, over-reliance on experts, difficulty in adapting to complex new network protocols, and inability to apply to actual network environments, the present invention proposes a design and implementation scheme for encrypted traffic anomaly detection based on high-performance traffic collection. Compared with existing encrypted traffic anomaly detection, this scheme will comprehensively consider the relationship between business characteristics and anomaly features, aiming to obtain more comprehensive, efficient, and accurate encrypted traffic information, thereby achieving applicability to high-speed network environments. The present invention implements support services for the entire process of traffic collection - business classification - anomaly detection, responds to traffic behavior anomalies of service equipment in a timely manner, and ensures the effectiveness and reliability of network and information system operations. At the same time, an incremental self-evolution strategy is used to achieve dynamic updates of portraits and models, ensuring the longevity of the life cycle of the anomaly detection function and ensuring continuous encrypted traffic supervision in real networks.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A design and implementation scheme for encrypted traffic anomaly detection based on high-performance traffic collection includes the following steps:

[0008] (1) Based on DPDK, high-performance traffic collection is performed to capture and collect high-speed traffic in network data centers. At the same time, the spatiotemporal features of the traffic are extracted in units of flow (a collection of data packets with the same source IP address, sink IP address, source port, sink port, and protocol), and the extracted spatiotemporal features are persisted in the database.

[0009] (2) To meet the specific business classification requirements for encrypted traffic, a two-stage technology is proposed to realize the system functions: automated online encrypted traffic collection and sample construction, and business classification model construction. The automated online encrypted traffic collection and sample construction part will sequentially perform automated sample collection and preliminary labeling, sample label construction, sample purification and feature extraction. Multi-level business classification based on heterogeneous features is divided into efficient business classification based on lightweight models, high-precision business classification based on deep learning models, and balanced business classification based on multi-model combination;

[0010] (3) Based on high-performance traffic collection and multi-level encrypted traffic business classification, we implement business-oriented anomaly detection and propose the construction of device-level traffic profiling. From the perspective of system database construction, we implement the spatiotemporal granularity division of traffic information and prediction results, and the spatiotemporal division of business classification. Based on the device-level traffic in the network scenario, we propose a variety of anomaly situations and corresponding evaluation strategies, and build a traffic anomaly detection model based on this to realize the anomaly detection function. At the same time, in response to the concept drift problem in the field of traffic analysis, we design a lifecycle maintenance method for device-level traffic profiling and business classification models based on the self-evolution strategy.

[0011] Furthermore, the step (1) specifically includes the following sub-steps:

[0012] (1.1) This paper designs a high-performance traffic collection system based on DPDK to capture and collect high-speed traffic in network data centers. It also extracts the spatiotemporal characteristics of traffic in units of flows (a collection of data packets with the same source IP address, destination IP address, source port, destination port, and protocol), providing a data basis for subsequent traffic analysis.

[0013] (1.2) Using DPDK's CPU affinity, several different tasks are bound to different CPUs to complete the acquisition program control, network data flow collection, network packet analysis, and network flow data persistence operations respectively:

[0014] (1.2.1) Control core: mainly responsible for controlling the acquisition program, collecting statistics of each CPU core in the acquisition program, and controlling the start and end of the acquisition program;

[0015] (1.2.2) Collection core: mainly responsible for data packet collection, mainly obtaining network traffic data packets from the network adapter's lock-free ring buffer queue (RX queue) in a polling manner;

[0016] (1.2.3) Analysis Core: This core is responsible for analyzing data packets. It extracts the five-tuple information of the data packet by parsing the various protocol layers of the data packet, and extracts the spatiotemporal characteristics of the traffic flow in units of flows.

[0017] (1.2.4) Writing core: Mainly responsible for the persistence of data packets, writing data packets in the form of pcap files, and writing the extracted flow features into the database regularly.

[0018] (1.3) The overall specific process of high-performance traffic collection and storage is as follows:

[0019] (1.3.1) The control core accepts command parameters, initializes the DPDK acquisition environment based on the command parameters, sets up the corresponding hardware in the system (such as CPU cores and memory thread pools), binds different tasks to the corresponding CPU cores, and starts the tasks;

[0020] (1.3.2) The acquisition core obtains network traffic packets from the network adapter's RX queue in a polling manner, receives packets in batches from the network card, and queues them into the corresponding lock-free ring buffer queues;

[0021] (1.3.3) The analysis core takes out data packets in batches from the RX queue, parses the protocol formats of each layer of the data packets, extracts the five-tuple information of the data packets, and determines whether the source IP address / destination IP address of the data packets already exists, thereby updating the spatiotemporal characteristics of the flow.

[0022] (1.3.4) The write core takes the data packet from the RX queue and writes it into the pcap file. At the same time, it periodically persists the device's flow feature table to the database.

[0023] Furthermore, the step (2) specifically includes the following sub-steps:

[0024] (2.1) Unlike high-performance traffic collection and storage, step (2) aims to provide effective label dataset support for the encrypted traffic service classification model without focusing on traffic collection performance. Due to the requirements of building a service classification model, the present invention includes the process of automated sample collection, manual intervention sample label construction, sample purification and feature extraction.

[0025] Furthermore, the step (2.1) specifically includes the following sub-steps:

[0026] (2.1.1) A network traffic collection and sample construction method based on the "PID (Process IDentifier)-port" pair is used to automatically extract the mapping relationship between the process PID and the network port within the time window, thereby achieving the correspondence between the process and the network traffic. In the network system scenario, the traffic of each process is isolated and preliminary samples are constructed.

[0027] (2.1.2) In view of the possible one-to-many mapping relationship between business applications and processes, the traffic samples of multi-process network services are associated through screening. Based on the obtained process traffic samples, the traffic between multiple processes of a single application is associated online to obtain complete pure business traffic samples, and a hash mapping with one-to-one correspondence between sample traffic and sample labels is established.

[0028] (2.1.3) Perform out-of-order recovery and retransmission deduplication, filtering and purification, and protocol data unit feature extraction on the originally collected encrypted traffic. At the same time, selectively remove weakly related traffic data such as ACK messages, handshake messages, and FIN messages, filter specific encrypted traffic with information such as fixed ports and fixed protocol header formats, retain highly sensitive features of business classification, create tables based on key values ​​such as target IP addresses, and store the data features of the collected traffic in the database.

[0029] (2.2) In order to adapt to the accuracy and efficiency of encrypted traffic anomaly detection in real scenarios, the present invention designs a multi-level business classification scheme based on heterogeneous features, fully utilizing the metadata features, statistical features, sequence features and other heterogeneous features of the collected sample traffic, and adopting traditional ensemble learning, deep learning and other classification models respectively to provide reliable traffic pattern support for the implementation of step (3). The output model will be connected to the database in step (1), and the collected encrypted traffic will be classified and predicted based on a fixed time interval, and the prediction results will be persisted in the database for subsequent anomaly detection queries.

[0030] Furthermore, the step (2.2) specifically includes the following sub-steps:

[0031] (2.2.1) In order to achieve high efficiency in business classification, the present invention constructs a lightweight business classification model based on the statistical features of sample flows. Based on the traffic metadata features of the business classification database and calculating the macro-statistical features, valuable features are extracted respectively, and flow feature vectors are uniformly constructed. The flow feature vectors are reduced and integrated using the principal component analysis method. The highly expressive flow feature vectors mined are used as input data, and representative integrated classification models such as random forest, Adaboost, and Xgboost are selected. The best one is selected as the lightweight business classification model based on the training results, ultimately achieving efficient and fast classification with acceptable accuracy.

[0032] (2.2.2) In order to achieve high accuracy in business classification, the present invention proposes to use the multi-protocol data unit length sequence as a feature, and to construct a length-sensitive deep learning model based on the feature extraction method of the sliding supersequence and N-gram model. The fine-grained sequence features with stronger representation capabilities are used as input, and the flow length sequence features are read from the business classification database as a sample data set, thereby establishing well-known sequence classification models such as LSTM, Fs-Net, and CapNet. The model is trained and tested in a deep learning manner, and the best one is selected as a high-precision business classification model based on the training results.

[0033] (2.2.3) In response to the comprehensive requirements of business classification indicators (ensuring high efficiency and high precision), the present invention comprehensively evaluates the advantages and disadvantages of lightweight classification models and high-precision classification models, and proposes a multi-model joint precision classification method to greatly improve the efficiency and reliability of classification results, and effectively reduce the risk and maintenance cost of subsequent anomaly detection. The present invention applies a lightweight classification model and a high-precision classification model in series, manually sets a threshold in the results of the lightweight classification model, regards the high-confidence results as the final prediction value, and re-inputs the data with confidence lower than the threshold into the high-precision classification model for secondary classification. Through manual debugging, the appropriate threshold is finally selected to achieve a balance between classification efficiency and accuracy.

[0034] Furthermore, the step (3) specifically includes the following sub-steps:

[0035] (3.1) Based on the device traffic table in the system database, traffic is segmented into given time slices and a device traffic profile is constructed. The specific process is as follows:

[0036] (3.1.1) The destination IP address is used as the basis for space division. The destination IP address is the specified host IP (the destination IP address is not necessarily the destination IP address in the traffic quintuple). For example, for all host IPs belonging to a certain subnet segment in the intranet, if the source IP or destination IP of the network flow is the same as the destination IP, the flow will be divided under this destination IP.

[0037] (3.1.2) After high-performance traffic collection, the time dimension is arranged according to the start timestamp of the flow, and the time period of the network flow is divided under the condition of setting the time slice size. In this way, the network traffic is divided into time and space dimensions.

[0038] (3.1.3) Traffic information tables for different devices are constructed using IP addresses as table names. Table attributes include underlying features such as flow timestamps, flow five-tuple information, flow length sequences, and various flow statistics. They also include labels predicted by the business model for each flow. Therefore, all entries in each table constitute a traffic profile for the corresponding IP device.

[0039] (3.2) Based on device-level traffic profile information, we propose multiple anomaly assessment strategies and summarize them into an anomaly detection mechanism, building a multi-strategy integrated anomaly detection model. By fully utilizing traffic metadata, statistical features, sequence features, and service category information, we achieve reliable and complete device-level anomaly detection. This detection mechanism is applied to system implementation, and when a specified anomaly is detected, early warning information is issued in a timely manner. The specific process is as follows:

[0040] (3.2.1) Traffic profiling-based anomalies differ from intrusion and malicious traffic in that they primarily explore the temporal and spatial distribution patterns of network traffic, and then, based on prior knowledge, provide early warnings of potential anomalies on a device. The following four potential anomalies are identified:

[0041] Abnormal device traffic scale: When a device fails, has high communication latency, or is attacked by a DDos, the device's network traffic scale will differ significantly from its prior rules, which is considered an abnormal device traffic scale.

[0042] Business label anomaly: By dynamically adjusting the confidence threshold, the flow with the highest confidence less than the confidence threshold is regarded as unknown business, and the "unknown business" label is added to it, and it is classified as a business label anomaly.

[0043] Abnormal business traffic distribution: Based on the predicted business label attributes in the device-level traffic profile, the business category distribution of each device in the designated time slice is analyzed. If the difference between the business category distribution and the prior distribution of the device exceeds the indicator range, it is considered to be abnormal business traffic distribution.

[0044] Business traffic time anomaly: Based on the predicted business tag attributes in the device-level traffic profile, the distribution of each business category of the device in the time dimension is analyzed. If the difference between the distribution and the prior distribution of the device exceeds the indicator range, it is considered a business traffic distribution anomaly.

[0045] (3.3) Based on the above potential anomalies, anomaly detection strategies are formulated for device traffic scale anomalies, service tag anomalies, service traffic distribution anomalies, service traffic time anomalies, etc., and each strategy is integrated to generate an asynchronous anomaly detection mechanism. In the system application, each strategy is executed and evaluated in parallel to improve the response speed of the anomaly detection process.

[0046] (3.3.1) For abnormal device traffic scale: Based on the IP device-level traffic profile table, set the time slice to build a sliding window, select data items through the sliding window, and analyze the selected data items with data analysis methods. Use the previous window information as prior information, and use indicators such as Euclidean distance and Manhattan distance to measure the difference between the current information and the prior information. Under the condition of setting a threshold, if the difference between the previous and the next information exceeds the threshold, it is considered that an abnormal device traffic scale has occurred, and an abnormal information will be prompted.

[0047] (3.3.2) For business tag anomalies: Based on the IP device-level traffic profile table, query the confidence of the business tag attributes. Under the condition of the set threshold, if a business tag with a confidence level lower than the threshold appears, it is considered a business tag anomaly and an anomaly information is prompted.

[0048] (3.3.3) Regarding abnormal business traffic distribution: Based on the IP device-level traffic profile table, count the number of each business tag in the current window, use the previous window information as prior information, and use indicators such as Mahalanobis distance and Euclidean distance to measure the difference between the current information and the prior information. Under the condition of a given threshold, if the difference between the previous and the next exceeds the threshold, it is considered that a business traffic distribution abnormality has occurred, and the system will prompt an abnormal information.

[0049] (3.3.4) For service traffic time anomalies: Based on the IP device-level traffic profile table, set a larger time slice, count the distribution of each service tag in the time slice, summarize the service traffic pattern using data analysis methods, and use the preceding window information as a priori information for comparison. If the traffic pattern of a certain service tag (differences from the prior information exceed an acceptable range), it is considered a service traffic time anomaly and an abnormality message is prompted.

[0050] The constructed device portrait and anomaly detection models inevitably face the problem of concept drift and cannot continuously achieve accurate business classification and anomaly detection. Therefore, the present invention proposes a lifecycle maintenance method for device portraits and detection models. Guided by the idea of ​​self-evolution, it continuously collects the latest business traffic samples and sets an available time period to update traffic samples and device traffic portraits, ultimately ensuring the continued effectiveness and availability of the traffic anomaly detection function.

[0051] Device-level traffic profiles are the data source for the anomaly detection mechanism of this invention. Therefore, periodic updates to these profiles are a crucial component of this invention. Therefore, this invention proposes discarding older device traffic profiles at fixed intervals and continuously supplementing them with newly collected profile information to ensure that the profile data in the database is more consistent with the current network status. Timely updates and maintenance of device-level traffic profiles improve overall system efficiency and model inference accuracy.

[0052] Since the anomaly detection mechanism of the present invention is mainly based on device-level traffic profiling, and the business tags therein are an important basis for anomaly detection, the present invention proposes to collect business traffic samples through automated online encrypted traffic collection and sample construction in a fixed time period, expand and update the original business data set, and use the incremental learning idea to achieve continuous learning of the business classification model while maintaining the learned features of the business classification model, thereby providing continuous and reliable advanced traffic features for business-oriented encrypted traffic anomaly detection.

[0053] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0054] (1) High-speed traffic collection provides real data for anomaly detection and is applicable to high-speed network environments. High-performance traffic collection is mainly based on DPDK for high-speed traffic collection, which can capture and collect high-speed traffic in network data centers. At the same time, it extracts the spatiotemporal characteristics of traffic based on flows (a collection of data packets with the same source IP address, sink IP address, source port, sink port, and protocol) and persists the extracted spatiotemporal characteristics in the traffic database, providing a data foundation for subsequent traffic analysis.

[0055] (2) Multi-level encrypted traffic service classification To meet the specific service classification requirements for encrypted traffic, a two-stage technology is proposed to realize the system functions: automated online encrypted traffic collection and sample construction, and service classification model construction. The automated online encrypted traffic collection and sample construction part will sequentially perform automated sample collection and preliminary labeling, sample label construction, sample purification and feature extraction. Multi-level service classification based on heterogeneous features is divided into efficient service classification based on lightweight models, high-precision service classification based on deep learning models, and balanced service classification based on multi-model combination.

[0056] (3) Business-oriented encrypted traffic anomaly detection is based on the realization of "high-performance traffic collection and storage method" and "multi-level encrypted traffic business classification". Business-oriented anomaly detection is realized on the basis of high-performance traffic collection and multi-level encrypted traffic business classification. Business-oriented encrypted traffic anomaly detection proposes the construction of device-level traffic profiles, realizes the spatiotemporal granularity division of traffic information and prediction results from the construction level of the system database, and realizes the spatiotemporal division of business classification results required by this project; proposes anomaly detection for business classification, proposes a variety of anomalies and corresponding evaluation strategies based on the device-level traffic in high-speed network scenarios, and builds a traffic anomaly detection model based on this, and systematically implements the anomaly detection function, and timely issues warnings for anomalies in online monitoring, meeting the anomaly detection needs of this project for business classification; in response to the concept drift problem in the field of traffic analysis, a life cycle maintenance method for device-level traffic profiles and business classification models is designed based on the self-evolution strategy.

[0057] (4) This patent implements encrypted traffic business classification and business-oriented encrypted traffic anomaly detection services in multiple demand scenarios through high-performance traffic collection and storage, multi-level encrypted traffic business classification, and business-oriented encrypted traffic anomaly detection. It provides support services for the entire process of traffic collection, business classification, and anomaly detection, and promptly responds to traffic behavior anomalies of service equipment to ensure the effectiveness and reliability of network and information system operations. At the same time, it uses an incremental self-evolution strategy to achieve dynamic updates of profiles and models, ensuring the longevity of the anomaly detection function and ensuring continuous encrypted traffic monitoring in high-speed networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is an encrypted traffic anomaly detection process based on high-performance traffic collection.

[0059] Specific implementation:

[0060] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0061] The present invention proposes a method for detecting anomalies of encrypted traffic based on high-performance traffic collection, which mainly consists of three parts: high-performance traffic collection and storage, multi-level encrypted traffic service classification, and business-oriented encrypted traffic anomaly detection. Among them, the first part is high-performance traffic collection and storage, which is mainly based on DPDK for high-speed traffic collection, to achieve the capture and collection of high-speed traffic in the network data center, extract features and persist them in the traffic database. The second part is multi-level encrypted traffic service classification, which proposes a two-stage technology of automated online encrypted traffic collection and sample construction, and business classification model construction to achieve specific business classification of encrypted traffic. The third part is business-oriented encrypted traffic anomaly detection. Based on the first and second parts, it proposes the construction of device-level traffic profiles. According to various abnormal situations and corresponding evaluation strategies of device-level traffic in real network scenarios, a traffic anomaly detection model is constructed based on this to achieve real-time anomaly warning. At the same time, in response to the concept drift problem in the field of traffic analysis, a life cycle maintenance method for device-level traffic profiles and business classification models is designed based on the self-evolution strategy.

[0062] Specifically, the present invention has the following steps:

[0063] (1) Based on DPDK, high-performance traffic collection and storage are performed to capture and collect high-speed traffic in network data centers. At the same time, the spatiotemporal characteristics of traffic are extracted based on flows (a collection of data packets with the same source IP address, sink IP address, source port, sink port, and protocol). The extracted spatiotemporal characteristics are persisted in the traffic database to provide a data foundation for subsequent traffic analysis.

[0064] The specific process of this step is as follows:

[0065] (1.1) Through DPDK's CPU affinity, control plane threads and data plane threads are bound to different CPU cores, avoiding the performance consumption of repeated scheduling between CPUs and fully utilizing the performance of multi-core CPUs. Therefore, in the traffic collection design, based on the actual needs of traffic collection and the characteristics of the DPDK framework, several different tasks are bound to different CPUs to respectively complete the operations of collection program control, network data flow collection, network packet analysis, and network flow data persistence;

[0066] (1.2) Using DPDK's CPU affinity, several different tasks are bound to different CPUs to complete the acquisition program control, network data flow acquisition, network packet analysis, and network flow data persistence operations.

[0067] (1.2.1) Control core: Mainly responsible for controlling the acquisition program, collecting statistics of each CPU core in the acquisition program (number of collected packets, number of bytes, packet loss rate, etc.), and controlling the start and end of the acquisition program;

[0068] (1.2.2) Collection Core: This core is responsible for collecting data packets in a round-robin fashion. This core retrieves network traffic packets from the network adapter's RX queue, which is the packet receive queue. After receiving the network traffic packets, it adds metadata to the data and uses the rte_ring_enqueue_burst() function to enqueue the processed data into the corresponding RX queue.

[0069] (1.2.3) Analysis Core: This core is responsible for analyzing data packets. It extracts the five-tuple information (source IP address, sink IP address, source port, sink port, and protocol) of the data packets by parsing the protocols at each layer of the data packets, and extracts the spatiotemporal characteristics of the traffic flow in units of flows.

[0070] (1.2.4) Writing core: Mainly responsible for the persistence of data packets, writing data packets in the form of pcap files, and writing the extracted flow features into the database regularly.

[0071] (1.3) The overall specific process of high-performance traffic collection and storage is as follows:

[0072] (1.3.1) First, the control core accepts the command parameters and initializes the DPDK acquisition environment EAL according to the command parameters. During the initialization of the EAL layer, the lcore will be initialized. In the DPDK function, the rte_eal_cpu_init() function will be called to obtain the device data in the corresponding file for the device information in the system, thereby obtaining information such as which CPU cores are available in the current system, and the parameters will be set through the eal_parse_args() function. At the same time, the control core will create a memory pool (rte_mempool), configure and start the network card parameters (determine whether the network card device is valid, configure and start the RX queue, turn on the promiscuous mode, etc.), bind the callback functions of different tasks to their respective CPU cores, and start the tasks;

[0073] (1.3.2) Secondly, the acquisition core obtains network traffic packets from the network adapter's RX queue in a polling manner, receives packets in batches from the network card, and queues them into the corresponding lock-free ring buffer receive queues (rx_ring);

[0074] (1.3.3) Subsequently, the analysis core extracts data packets from rx_ring in batches, parses the protocol format of each layer of the data packet, extracts the five-tuple information of the data packet, and determines whether the source IP address / destination IP address of the data packet already exists: if not, a new device IP feature flow table is created and the spatiotemporal features of the flow are added; if already exists, the flow of the data packet is checked to see if it already exists in the device IP feature flow table: if not, the spatiotemporal features of the flow are added; if so, the spatiotemporal features of the flow are updated.

[0075] (1.3.4) The write core takes data packets from rx_ring and writes them into the pcap file. At the same time, it periodically persists the device's flow feature table into the database.

[0076] (2) Aiming at the specific business classification of encrypted traffic in large-scale network scenarios, a two-stage technology of automated online encrypted traffic collection and sample construction, and business classification model construction is proposed to realize encrypted traffic business classification. The present invention successively realizes automated online encrypted traffic collection and sample construction, and multi-level business classification based on heterogeneous features. Among them, the automated online encrypted traffic collection and sample construction part will sequentially perform automated sample collection and preliminary marking, sample label construction, sample purification and feature extraction; the multi-level business classification based on heterogeneous features is divided into efficient business classification based on lightweight models, high-precision business classification based on deep learning models, and balanced business classification based on multi-model combination.

[0077] The specific process of this step is as follows:

[0078] (2.1) Unlike the high-performance traffic collection and storage in the first part, the present invention aims to provide effective label dataset support for the encrypted traffic service classification model without focusing on traffic collection performance. Therefore, the steps of automatic sample collection, manual intervention sample label construction, sample purification and feature extraction are designed.

[0079] (2.1.1) In order to solve the problem of background traffic interference in the collection of traffic label samples, a network traffic collection and sample construction method based on the "PID (Process IDentifier)-port" pair is implemented. This method breaks through the limitation of traditional methods that require a pure application running environment. By automatically extracting the mapping relationship between the process PID and the network port within the time window, the correspondence between the process and the network traffic is achieved. In the real network system scenario, the traffic of each process is isolated and preliminary samples are constructed.

[0080] (2.1.2) To address the potential one-to-many mapping relationships between business applications and processes, we will implement traffic sample correlation for multi-process network services through post-process manual screening. Based on the previously acquired process traffic samples, we will perform online correlation between traffic flows across multiple processes within a single application to obtain complete, pure business traffic samples. Ultimately, we will achieve the collection of labeled traffic samples based on business categories, further labeling the samples with business labels using controllable devices, and establishing a one-to-one hash mapping between sample traffic and sample labels.

[0081] (2.1.3) Perform out-of-order recovery and retransmission deduplication, filtering and purification, and protocol data unit feature extraction operations on the originally collected encrypted traffic. For example, to address the errors between the spliced ​​traffic and the original network data of the application caused by TCP out-of-order and message retransmission, adopt the solution of TCP out-of-order reordering of the original traffic and deduplication of the retransmitted messages to better restore the business data characteristics. At the same time, selectively clear weakly related traffic data such as ACK messages, handshake messages and FIN messages, filter specific encrypted traffic with information such as fixed ports and fixed protocol header formats, and retain highly sensitive characteristics of business classification. Finally, store the traffic data characteristics generated by different devices in blocks, create tables based on key values ​​such as target IP addresses, and store the traffic data characteristics collected by different devices in the business classification database, providing a data foundation for establishing a highly robust business classification model in the next stage.

[0082] (2.2) Considering the need for accurate and efficient encrypted traffic anomaly detection in real-world scenarios, the present invention designs a multi-level service classification technology based on heterogeneous features. This technology fully utilizes heterogeneous features such as metadata features, statistical features, and sequence features of the collected sample traffic, and adopts classification models such as traditional ensemble learning and deep learning. On the basis of meeting the needs of multi-level service classification, it also provides reliable traffic pattern support for the implementation of the third part of anomaly detection. At the same time, the multi-level service classification model will be connected to the first part of the database, and the collected encrypted traffic will be classified and predicted based on fixed time intervals. The prediction results will be persisted in the database for subsequent anomaly detection queries.

[0083] The specific process of this step is as follows:

[0084] (2.2.1) In response to the high efficiency requirements of business classification, the present invention constructs a lightweight business classification model based on the statistical features of sample flows. Based on the traffic metadata features of the business classification database and the calculation of macro-statistical features, valuable features such as the interval between adjacent messages, the number of bytes transmitted per second, flow duration, flow active time, and the number of uplink and downlink messages are extracted. A flow feature vector is uniformly constructed and reduced and integrated using the principal component analysis method. The highly expressive flow feature vector mined is used as input data. Representative integrated classification models such as random forest, Adaboost, and Xgboost are selected and selected as lightweight business classification models based on the training results, thereby achieving efficient and rapid classification with acceptable accuracy and improving the overall efficiency of the entire process of all stages of anomaly detection.

[0085] (2.2.2) In response to the high-precision requirements of business classification, the present invention proposes to use the multi-protocol data unit length sequence as a feature, and based on the feature extraction method of the sliding supersequence and N-gram model, construct a length-sensitive deep learning model. The fine-grained sequence features with stronger representation capabilities are used as input, and the flow length sequence features are read from the business classification database as a sample data set. Well-known sequence classification models such as LSTM, Fs-Net, and CapNet are established. The model is trained and tested in a deep learning manner, and the best one is selected as a high-precision business classification model based on the training results.

[0086] (2.2.3) In response to the comprehensive requirements of business classification indicators, the present invention comprehensively evaluates the advantages and disadvantages of lightweight classification models and high-precision classification models, and proposes a multi-model joint precision classification method to significantly improve the efficiency and reliability of classification results, and effectively reduce the risk and maintenance cost of subsequent anomaly detection. The present invention applies a lightweight classification model and a high-precision classification model in series, sets a threshold in the results of the lightweight classification model, regards the high-confidence results as the final prediction value, and re-inputs the data with confidence below the threshold into the high-precision classification model for secondary classification. By debugging and selecting the appropriate threshold, a balance between classification efficiency and accuracy is achieved.

[0087] (3) Business-oriented encrypted traffic anomaly detection: Based on high-performance traffic collection and multi-level encrypted traffic service classification, this paper proposes the construction of device-level traffic profiling. This achieves the spatiotemporal granularity of traffic information and prediction results from the perspective of system database construction. Based on various abnormal situations and corresponding evaluation strategies of device-level traffic in real network scenarios, a traffic anomaly detection model is constructed to achieve online anomaly detection. In addition, to address the concept drift problem in the field of traffic analysis, this paper designs a lifecycle maintenance method for device-level traffic profiling and service classification models based on a self-evolution strategy, fully meeting all aspects of business-oriented traffic anomaly detection.

[0088] The specific process of this step is as follows:

[0089] (3.1) Divide the traffic data and business classification results into time and space dimensions, and build a device-level traffic profile based on this. The traffic profile fully describes the traffic flow through the target IP device, and then reflects the device's operating status from a traffic perspective, thereby providing reliable evaluation information for the next stage of anomaly detection tasks.

[0090] (3.1.1) First, the target IP address is used as the basis for space division. The target IP address is the designated host IP of interest (the target IP address is not necessarily the destination IP address in the traffic quintuple). For example, for all host IPs belonging to a certain subnet segment in the intranet, if the source IP or destination IP of the network flow is the same as the target IP, the flow will be divided under this target IP.

[0091] (3.1.2) Secondly, after high-performance traffic collection, the time dimension is arranged according to the start timestamp of the flow. Under the condition of manually setting the time slice size, the network traffic division in the time and space dimensions is completed, and the device-level traffic profile is constructed based on the IP division. Specifically, the present invention constructs the traffic information table of different devices with IP as the table name. The attributes of the table include underlying features such as flow timestamp, flow five-tuple information, flow length sequence, and various flow statistical features. It also contains labels predicted by the business model for each flow. Therefore, all the table item information of each table constitutes the traffic profile of the corresponding IP device.

[0092] (3.2) Based on device-level traffic profile information, the present invention proposes a variety of anomaly assessment strategies, which are summarized into an anomaly detection mechanism, and constructs a multi-strategy integrated anomaly detection model. Ultimately, based on the full utilization of traffic metadata information, statistical features, sequence features, and business category information, reliable and complete device-level anomaly detection is achieved, and early warning information is issued in a timely manner when a specified anomaly is detected.

[0093] (3.2.1) Traffic profiling-based anomalies are different from intrusion traffic and malicious traffic. They mainly explore the distribution patterns of network traffic in the time and space dimensions, and then use prior knowledge to warn of possible anomalies of a certain device. Since the device-level traffic profiling records the traffic information of different IP addresses in the time dimension, anomalies in coarse-grained time periods (longer periods of time) and fine-grained time periods (shorter periods of time) can be dynamically analyzed based on this information. The present invention sets the following potential anomalies:

[0094] Abnormal device traffic scale: Device-level traffic profiling records detailed traffic information for each IP device in chronological order. The total amount of network traffic generated by each device within a fixed time period (such as a day) should remain relatively stable. By manually setting time slices, we analyze the traffic information of each IP in different time slices and compare the differences in the time dimension. Therefore, when a device fails, communication latency is high, or it is attacked by DDos, the network traffic scale of the device will be significantly different from the a priori pattern, and this is considered an abnormal device traffic scale.

[0095] Service label anomalies: Each encrypted flow in the device-level traffic profile will include a predicted service label after being predicted by the "Multi-Level Encrypted Traffic Service Classification" module. The multi-classification model generates confidence scores for each service category for that flow, and the model uses the category with the highest confidence score as the prediction. Therefore, by manually and dynamically adjusting the confidence threshold, flows with a maximum confidence score below the confidence threshold are considered unknown services and are labeled "Unknown Service." Based on prior knowledge of different devices, it is possible to determine whether the device profile for the current time slice exhibits service label anomalies.

[0096] Abnormal business traffic distribution: Based on the predicted business label attributes in the device-level traffic profile, the business category distribution of each device in the designated time slice can be analyzed. When the evaluation index is set, if the difference between the distribution and the prior distribution of the device exceeds the index range, it is considered to be abnormal business traffic distribution.

[0097] Business traffic time anomaly: Based on the predicted business tag attributes in the device-level traffic profile, the distribution of each business category of the device in the time dimension can be analyzed. When the evaluation index is set, if the difference between it and the prior distribution of the device exceeds the index range, it is considered a business traffic distribution anomaly.

[0098] (3.2.2) Based on the above potential anomalies, anomaly detection strategies are formulated for device traffic scale anomalies, service tag anomalies, service traffic distribution anomalies, service traffic time anomalies, etc., and each strategy is integrated to generate an asynchronous anomaly detection mechanism. Each strategy is executed in parallel for evaluation, thereby improving the response speed of the anomaly detection process. The specific anomaly detection strategies of the present invention are as follows:

[0099] For abnormal device traffic scale: Based on the system database, query the IP device-level traffic profile table separately, set multiple time slice scales (such as 1 hour, 12 hours, 24 hours), and construct sliding windows of corresponding scales. Sort the table items by timestamp, and select data items with a sliding window. Use data analysis methods to analyze the total number of bytes, total number of packets, and other information of the selected network flow. Use the preceding window information as prior information, and use indicators such as Euclidean distance and Manhattan distance to measure the difference between the current information and the prior information. Under the condition of setting a threshold, if the difference between the previous and the next exceeds the threshold, it is considered that the device traffic scale is abnormal, and the system will prompt the abnormal information.

[0100] For business tag anomalies: Based on the system database, query the IP device-level traffic profile table separately, and query the confidence of the business tag attributes. Under the condition of the set threshold, if a business tag with a confidence level lower than the threshold appears, it is considered a business tag anomaly and the system prompts the anomaly information.

[0101] For abnormal business traffic distribution: Based on the system database, query the IP device-level traffic portrait table separately, count the number of each business label in the current window separately, use the previous window information as prior information, and use indicators such as Mahalanobis distance and Euclidean distance to measure the difference between the current information and the prior information. Under the condition of a given threshold, if the difference between the previous and the next exceeds the threshold, it is considered that a business traffic distribution abnormality has occurred, and the system will prompt the abnormal information.

[0102] For business traffic time anomalies: Based on the system database, query the IP device-level traffic portrait table separately, set a larger time slice scale (such as 24 hours), and count the distribution of each business tag in the time slice separately. Use data analysis methods to summarize the business traffic pattern, and use the preceding window information as prior information for comparison. If the traffic pattern of a certain business tag (such as business traffic peak period, idle period) differs from the prior information by more than an acceptable range, it is considered that a business traffic time anomaly has occurred, and the system will prompt an abnormality information.

[0103] (3.3) Due to the temporal differences of real network traffic in time and space dimensions, the device portraits and anomaly detection models constructed in the early stage inevitably face the problem of concept drift, which in turn leads to a short life cycle of the classification model and anomaly detection mechanism, and the inability to continuously achieve accurate business classification and anomaly detection. The present invention proposes a method for maintaining the life cycle of device portraits and detection models. Guided by the idea of ​​self-evolution, it continuously collects and persists the latest business traffic samples, manually sets the available time period of the portraits and models, and eliminates old traffic samples and device traffic portraits based on this to maintain the periodic update of data and portraits. At the same time, the automatic iteration of the model is completed based on the time period, ultimately ensuring the continued effectiveness and availability of the traffic anomaly detection function. Therefore, the present invention significantly reduces the risk of model unavailability caused by the concept drift problem by automatically updating the model, thereby reducing the product maintenance cost to the greatest extent, reflecting the characteristics of the present invention of high quality, low risk and easy maintenance.

[0104] (3.3.1) The device-level traffic profile is the data source of the anomaly detection mechanism of the present invention, so the periodic update of the device traffic profile is an important part of the present invention. In a real network scenario, as time goes by, the device traffic profile in the system database will expand rapidly. On the one hand, this will take up a lot of storage space, resulting in a decrease in query efficiency, which in turn affects the efficiency of the whole process reasoning. On the other hand, due to the Markov property of traffic data, the device traffic profile in the present invention is often only associated with the first N windows (N can be dynamically adjusted according to demand). Therefore, it is proposed to discard the early device traffic profile in a fixed time period and continuously supplement the newly collected profile information to make the profile data in the database more in line with the current network reality. Real state.

[0105] (3.3.2) Since the anomaly detection mechanism of the present invention is mainly based on device-level traffic profiling, and the business label therein is an important basis for anomaly detection, the accuracy of the data attributes largely determines the anomaly detection effect. Therefore, the self-evolution of business classification data and models is an important part of maintaining the cycle of this project. The present invention proposes to collect business traffic samples through automated online encrypted traffic collection and sample construction in a fixed time period, expand and update the original business data set, and use the idea of ​​incremental learning to increase the model's feature recognition of new sample data while maintaining the learned features of the business classification model, thereby realizing continuous learning of the business classification model and providing continuous and reliable advanced traffic features for business-oriented encrypted traffic anomaly detection.

[0106] The technical means disclosed in the solutions of the present invention are not limited to those disclosed in the above-mentioned embodiments, but also include technical solutions composed of any combination of the above-mentioned technical features. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for detecting anomalies in encrypted traffic based on high-performance traffic collection, characterized in that: The steps include: (1) High-performance traffic collection based on DPDK enables the capture and collection of high-speed traffic, while extracting traffic features in units of flow and persisting them in the traffic database; (2) Based on the business classification of encrypted traffic, a two-stage technology of automated online encrypted traffic collection and sample construction, and business classification model construction is proposed to achieve specific business classification for encrypted traffic; (3) Business-oriented encrypted traffic anomaly detection builds a traffic anomaly detection model based on various anomalies of device-level traffic in real network environments and corresponding evaluation strategies, and issues early warnings for anomalies in a timely manner; Wherein, step (1) specifically includes the following sub-steps: (1.1) Use the CPU control core to accept command parameters, initialize the DPDK acquisition environment based on the command parameters, bind different tasks to their respective CPU cores, and start the tasks; (1.2) The CPU acquisition core obtains network traffic data packets from the network card in a polling manner and sends them to each corresponding lock-free ring buffer RX queue; (1.3) Use the CPU analysis core to batch extract data packets from the RX queue, parse the protocol format of each layer of the data packet, extract the five-tuple information and corresponding features of the data packet, and update the corresponding features of the flow based on whether the flow already exists; (1.4) Use the CPU write core to take the data packet from the RX queue and write it to the pcap file. At the same time, the traffic characteristics of the device are persisted in the traffic database at regular intervals; The specific requirements of step (2) are: (2.1) The automated online encrypted traffic collection and sample construction section will sequentially perform automated sample collection and preliminary labeling, sample label construction, sample purification, and feature extraction; (2.2) Automated Sample Collection and Preliminary Labeling: By automatically extracting the mapping relationship between process PIDs and network ports within the time window, the correspondence between processes and network traffic is achieved, and the isolated traffic collection and preliminary sample construction of each process are completed; (2.3) Business category-oriented sample labeling: Based on the process traffic samples, the traffic between multiple processes of a single application business is online correlated to achieve sample labeling, and a hash map with a one-to-one correspondence between traffic samples and sample labels is established; (2.4) Sample purification and feature extraction implement out-of-order recovery, retransmission deduplication, filtering and purification, and protocol feature extraction operations. At the same time, it selectively removes weakly related traffic data, filters specific encrypted traffic with fixed information, retains highly sensitive features of business classification, and creates a table based on the target IP address key value classification to store the traffic sample data features collected by the corresponding device; (2.5) Multi-level service classification based on heterogeneous features is divided into efficient service classification based on lightweight models, high-precision service classification based on deep learning models, and balanced service classification based on multi-model combination. (2.6) Construct a lightweight service classification model based on the statistical characteristics of traffic samples. Statistical features are calculated based on the traffic metadata characteristics of the service classification database. Principal component analysis is used to reduce and integrate these features. The optimal classification results are selected using ensemble learning as the lightweight service classification model to achieve efficient and rapid classification with acceptable accuracy. (2.7) Using multi-protocol data unit length sequences as features, and based on the feature extraction method of sliding supersequence and N-gram model, a length-sensitive deep learning model is constructed using LSTM, Fs-Net, and CapNet sequence classification models. The model is trained and tested using deep learning, and the best model is selected based on the training results as a high-precision service classification model. (2.8) Comprehensively evaluate the strengths and weaknesses of lightweight and high-precision classification models, and propose a multi-model joint precision classification method to significantly improve the efficiency and reliability of classification results and effectively reduce the risk and maintenance costs of subsequent anomaly detection. The solution applies the lightweight and high-precision classification models in series, manually setting a threshold in the lightweight classification model results, treating high-confidence results as the final prediction value, and re-inputting data with confidence levels below the threshold into the high-precision classification model for secondary classification. The step (3) specifically includes the following steps: (3.1) Based on the traffic data and business classification results, the traffic is divided into time and space dimensions. On this basis, a device-level traffic profile is constructed. The traffic profile fully describes the traffic flow through the target IP device, and then reflects the operating status of the device from the traffic perspective, thereby providing reliable judgment information for the next stage of anomaly detection tasks; (3.2) Based on device-level traffic profile information, we propose multiple anomaly assessment strategies and aggregate them into an anomaly detection mechanism. We then build a multi-strategy integrated anomaly detection model. This model fully utilizes traffic metadata, statistical features, sequence features, and service category information to achieve reliable and complete device-level anomaly detection, and promptly issues warning alerts when a specified anomaly is detected. (3.3) Through the lifecycle maintenance method of device-level traffic profiling and anomaly detection models, guided by the concept of self-evolution, the latest business traffic samples are continuously collected and persisted. The available time period of the profiling and model is manually set. Based on this, old traffic samples and device-level traffic profiling are eliminated to maintain the periodic update of traffic samples and device-level traffic profiling. At the same time, the model is automatically iterated based on the time period to ensure the continued effectiveness and availability of the traffic anomaly detection function.

2. The method for detecting anomalies of encrypted traffic based on high-performance traffic collection according to claim 1, characterized in that: In step (1), Control core: responsible for controlling the acquisition program, collecting statistics of each CPU core in the acquisition program, and controlling the start and end of the acquisition program; Collection core: responsible for data packet collection, obtaining network traffic data packets from the network adapter's lock-free ring buffer queue (RX queue) in a polling manner; Analysis core: Responsible for data packet analysis. It extracts the five-tuple information of the data packet by parsing the various protocol layers of the data packet, and extracts the spatiotemporal characteristics of the traffic flow in units of flows. Writing core: responsible for the persistence of data packets, writing data packets in the form of pcap files, and writing the extracted traffic features into the traffic database at regular intervals.

3. The method for detecting anomalies of encrypted traffic based on high-performance traffic collection according to claim 2, characterized in that: The overall specific process of high-performance traffic collection and storage is as follows: The control core accepts command parameters, initializes the DPDK acquisition environment based on the command parameters, sets up the corresponding hardware in the system, binds different tasks to the corresponding CPU cores, and starts the tasks; The acquisition core obtains network traffic packets from the RX queue of the network adapter in a polling manner, receives packets in batches from the network card, and queues them into the corresponding lock-free ring buffer queues; The analysis core takes out data packets in batches from the RX queue, parses the protocol formats of each layer of the data packet, extracts the five-tuple information of the data packet, and determines whether the source IP address / destination IP address of the data packet already exists, thereby updating the spatiotemporal characteristics of the flow. The write core takes data packets from the RX queue and writes them into the pcap file, while periodically persisting the device's traffic characteristics into the traffic database.

4. The method for detecting anomalies of encrypted traffic based on high-performance traffic collection according to claim 2, characterized in that: The process of automated sample collection and preliminary labeling, sample label construction, sample purification and feature extraction includes the following sub-steps: The network traffic collection and sample construction method based on "PID-port" pairs automatically extracts the mapping relationship between the process PID and the network port within the time window, realizes the correspondence between the process and the network traffic, and collects the traffic of each process in isolation and constructs preliminary samples in the real end system scenario. Aiming at the one-to-many mapping relationship between business applications and processes, we implement the correlation of traffic samples of multi-process network services through screening. Based on the process traffic samples, we correlate the traffic between multiple processes of a single application online to obtain complete pure business traffic samples, and establish a hash mapping between sample traffic and sample labels. The original collected encrypted traffic is subjected to out-of-order recovery, retransmission deduplication, filtering and purification, and protocol feature extraction operations. At the same time, the weakly related traffic data of ACK messages, handshake messages, and FIN messages are selectively cleared, and specific encrypted traffic is filtered with fixed port and fixed protocol header format information. The highly sensitive characteristics of business classification are retained, and tables are created by target IP address key value classification. The collected traffic characteristics are stored in the traffic database.

5. The method for detecting anomalies of encrypted traffic based on high-performance traffic collection according to claim 2, characterized in that: In step (2), In order to improve the efficiency of business classification, a lightweight business classification model is constructed based on the statistical characteristics of traffic samples. The statistical characteristics are calculated based on the traffic metadata characteristics of the business classification database, and a unified traffic feature vector is constructed. The principal component analysis method is used to reduce and integrate the mined highly expressive flow feature vectors as input data. Random forest, Adaboost, and Xgboost ensemble classification models are selected and the best one is selected as the lightweight business classification model based on the training results, ultimately achieving efficient and fast classification with acceptable accuracy. To achieve high-precision service classification, we use multi-protocol data unit length sequences as features and construct a length-sensitive deep learning model based on the feature extraction method of sliding supersequence and N-gram model. We use fine-grained sequence features with stronger representation capabilities as input and read flow length sequence features from the service classification database as a sample data set to establish LSTM, Fs-Net, and CapNet sequence classification models. We complete model training and testing with deep learning, and select the best model based on the training results as a high-precision service classification model. In response to the comprehensive requirements of business classification indicators, the advantages and disadvantages of lightweight classification models and high-precision classification models are comprehensively evaluated, and a multi-model joint precision classification method is proposed to greatly improve the efficiency and reliability of classification results, and effectively reduce the risk and maintenance cost of subsequent anomaly detection. The lightweight classification model and the high-precision classification model are applied in series, and the threshold is manually set in the results of the lightweight classification model. The high-confidence results are regarded as the final prediction value, and the data with confidence lower than the threshold is re-input into the high-precision classification model for secondary classification. The threshold is finally selected through manual debugging to achieve a balance between classification efficiency and accuracy.

6. The method for detecting anomalies of encrypted traffic based on high-performance traffic collection according to claim 4, characterized in that: The specific process of step (3.1) is as follows: (3.1.1) The target IP address is used as the basis for space division. The target IP address is the designated host IP. If the source IP or sink IP of the network traffic is the same as the destination IP, the traffic will be divided into this destination IP. (3.1.2) After high-performance traffic collection, the time dimension is arranged according to the start timestamp of the flow. Under the condition of setting the time slice size, the time period of the network flow is completed. In this way, the network traffic is divided into time and space dimensions. (3.1.3) Build traffic information tables for different devices using IP as the table name. The table attributes include flow timestamp, flow five-tuple information, flow length sequence, and various flow statistical features. It also contains the label predicted by the service classification model for each flow. Therefore, all the table entries in each table constitute the traffic profile of the corresponding IP device. (3.2) Based on device-level traffic profile information, we propose multiple anomaly assessment strategies and summarize them into an anomaly detection mechanism. We then build a multi-strategy integrated anomaly detection model. This model fully utilizes traffic metadata, statistical features, sequence features, and service category information to achieve reliable and complete device-level anomaly detection. This detection mechanism is then applied to system implementation, and early warning information is issued in a timely manner when a specified anomaly is detected. The specific process is as follows: (3.2.1) Traffic profiling-based anomalies differ from intrusion traffic and malicious traffic. By mining the temporal and spatial distribution patterns of network traffic, we can provide early warnings of device anomalies based on prior knowledge. We identify four potential anomalies: Device traffic scale abnormality: When a device fails, communication delay is high, or it is attacked by DDos, the network traffic scale of the device will be significantly different from its prior rules, which is considered to be a device traffic scale abnormality. Business label anomaly: By dynamically adjusting the confidence threshold, the flow with the highest confidence less than the confidence threshold is regarded as unknown business, and the "unknown business" label is added to it, and it is classified as business label anomaly. Abnormal business traffic distribution: Based on the predicted business label attributes in the device-level traffic profile, the business category distribution of each device in the set time slice is analyzed. If the difference between the business category distribution and the prior distribution of the device exceeds the index range, it is considered to be abnormal business traffic distribution. Business traffic time anomaly: Based on the predicted business label attributes in the device-level traffic profile, the distribution of each business category of the device in the time dimension is analyzed. If the difference between the distribution and the prior distribution of the device exceeds the indicator range, it is considered a business traffic time anomaly.

7. The method for detecting anomalies of encrypted traffic based on high-performance traffic collection according to claim 6, characterized in that: For abnormal device traffic scale: according to the device-level traffic portrait, set the time slice to build a sliding window, select data items through the sliding window, and analyze the selected data items with the data analysis method. The previous window information is used as the prior information, and the Euclidean distance and Manhattan distance indicators are used to measure the difference between the current information and the prior information. Under the condition of the set threshold, if the difference between the previous and the next exceeds the threshold, it is considered that the device traffic scale is abnormal, and the abnormal information is prompted. For business tag anomalies: Based on the device-level traffic profile table, query the confidence of the business tag attributes. Under the condition of the set threshold, if a business tag with a confidence lower than the threshold appears, it is considered a business tag anomaly and an abnormality message is prompted. For abnormal business traffic distribution: Based on the device-level traffic profile, the number of each business tag in the current window is counted, the previous window information is used as prior information, and the Mahalanobis distance and Euclidean distance indicators are used to measure the difference between the current information and the prior information. Under the condition of a given threshold, if the difference between the previous and the next exceeds the threshold, it is considered that the business traffic distribution is abnormal, and the system prompts the abnormal information. For business traffic time anomalies: Based on the device-level traffic profile, set a larger time slice, count the distribution of each business tag in the time slice, summarize the business traffic pattern with data analysis methods, and use the preceding window information as prior information for comparison. If the traffic pattern of the business tag differs from the prior information by more than an acceptable range, it is considered a business traffic time anomaly and an abnormality message is prompted.

Images