Method and system for detecting unknown encrypted malicious traffic
By constructing realistic attack scenarios in specific network environments, generating encrypted malicious traffic datasets and preprocessing them, and building traffic interaction graphs and multi-dimensional loss functions to identify unknown encrypted malicious traffic, this method solves the problem of detecting unknown encrypted malicious traffic in existing technologies and achieves efficient and accurate detection results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-03-10
AI Technical Summary
Existing technologies are insufficient to effectively detect unknown encrypted malicious traffic, rely on prior knowledge of known attacks, have poor real-time performance, and cannot identify new malicious patterns in encrypted traffic.
We construct realistic attack scenarios in specific real-time network environments, generate an initial encrypted malicious traffic dataset, construct a traffic interaction graph through short-flow and long-flow data preprocessing, extract connected components and perform OPTICS clustering, and identify abnormal traffic by combining the KD-Tree algorithm and a multi-dimensional loss function.
It achieves efficient identification of unknown encrypted malicious traffic, improves detection coverage and scenario adaptability, reduces false positive rate, and meets the detection efficiency requirements of real-time network environment.
Smart Images
Figure CN121644148A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of network space security, and particularly relates to a detection method and system for unknown encrypted malicious traffic. BACKGROUND
[0002] With the deep penetration of digital infrastructure and the widespread enhancement of network communication security awareness, encryption technology has become a standard configuration of Internet traffic. Although this trend has significantly enhanced user privacy and data security, it has also provided a natural cover for network threat actors. Malware such as ransomware, advanced persistent threats, and botnets increasingly use encrypted channels for command and control communication, data leakage, and malicious payload delivery, effectively evading traditional security mechanisms that rely on deep packet inspection content.
[0003] In-depth analysis and detection of encrypted network traffic aims to penetrate the surface of the encryption layer and reveal potential malicious behavior patterns such as communication timing, traffic size, packet sequence, certificate characteristics, and protocol handshake anomalies, which are crucial for identifying and preventing malicious activities, containing their spread, and mitigating their destructive power.
[0004] In 2021, Nabil Seddigh's team proposed a high-speed network classification platform solution integrating DPI and semi-supervised machine learning, supporting binary encrypted data protocol / multi-type encrypted application classification and 10Gbps encrypted data high-accuracy identification, but only introduced the framework mechanism without mentioning the data analysis principle and high-speed network test effect; in 2022, researchers from the Institute of Information Engineering, Chinese Academy of Sciences proposed the ET-BERT traffic representation model, combining NLP pre-training technology with traffic data characteristics to learn implicit relationships of unlabeled traffic and improve traffic classification effect in different scenarios, but cannot identify unknown traffic; in 2024, Fu Chuanpu et al. proposed an unsupervised graph learning method, which reduces graph processing overhead and extracts structured features to distinguish malicious traffic by constructing traffic interaction graphs and combining short flow aggregation technology, but has limited effect on encrypted malicious traffic detection in specific real-time networks.
[0005] Attackers are constantly developing new variants of encrypted malicious traffic and using evasion techniques to bypass existing detection systems based on known features or signatures. Therefore, research on detection techniques for unknown encrypted malicious traffic is urgent, which continuously updates and optimizes the detection system by learning and adapting to new attack patterns in a timely manner, i.e., identifying new types of malicious traffic that have not been explicitly labeled or characterized, to ensure the balance between the confidentiality of legitimate communication and the maintenance of network space security. SUMMARY
[0006] To solve the above problems in the prior art, the application realizes the following technical solutions: The application discloses a method for detecting unknown encrypted malicious traffic. A real attack scene is constructed in a specific real-time network environment, and an initial encrypted malicious traffic dataset is generated. Short flow data and long flow data in the initial encrypted malicious traffic dataset are preprocessed respectively. A traffic interaction graph is constructed based on the processed short flow data and the processed long flow data. Connected components in the traffic interaction graph are extracted, and the connected components are clustered to obtain normal components and abnormal components. Suspected abnormal components are obtained by searching and clustering the abnormal components. A loss function in the process of obtaining the suspected abnormal components is calculated, and when the loss function is greater than a set threshold T, the suspected abnormal components are determined as encrypted malicious traffic.
[0007] The malicious encrypted traffic dataset includes encrypted flood traffic, encrypted malicious network traffic and encrypted traffic generated by malicious software.
[0008] The short flow data in the initial encrypted malicious traffic dataset is preprocessed, including: Short flow data meeting a constraint condition is aggregated and its features are retained to obtain short flow aggregation edges and features of the short flow aggregation edges, namely the processed short flow data.
[0009] The long flow data in the initial encrypted malicious traffic dataset is preprocessed, including: Continuous long flow data segments are obtained according to break points of the long flow data, and the continuous long flow data segments are classified into corresponding intervals according to lengths to obtain a long flow data histogram; Long flow feature edges and features of the long flow feature edges, namely the processed long flow data, are obtained according to the long flow data histogram, a total byte number of the long flow data and a flow duration of the long flow data.
[0010] Connected components in the traffic interaction graph are extracted, and the connected components are clustered into normal components and abnormal components, including: The traffic interaction graph is processed by using a depth-first search method to obtain connected components and feature dimensions of the connected components; Based on the feature dimensions of the connected components, the connected components are clustered by using an OPTICS clustering algorithm to obtain the normal components and the abnormal components.
[0011] Suspected abnormal components are obtained by searching and clustering the abnormal components, including: Features of short flow aggregation edges in the abnormal components and features of long flow feature edges are normalized by using minimum-maximum normalization to obtain normalized features. While rapidly searching the normalized features using the KD-Tree algorithm, clustering is performed based on the OPTICS clustering algorithm to obtain suspicious anomaly components.
[0012] The loss function obtained in the process of obtaining suspicious anomaly components includes:
[0013]
[0014]
[0015]
[0016] in, Indicates cluster center difference loss; A single cluster center is represented by the mean of all edge features within the cluster; Let L represent the set of all cluster centers, where L is the total number of clusters; f(e) is the feature vector of the edge. L2 norm is used to calculate the relationship between the eigenvector f(e) and the cluster centers. The straight-line distance between them; min indicates taking the minimum value; Indicates cluster time span loss; This represents the set of edge clusters generated by clustering, i.e., the cluster of edges of the same kind containing edge e; TimeRange() represents the time span operator; This indicates cluster flux density loss; Represents the edge cluster set Size; This indicates the total loss.
[0017] In a second aspect, the present invention provides a detection system for unknown encrypted malicious traffic, including a data acquisition module, a data preprocessing module, an interaction graph construction module, a first clustering module, a second clustering module, and a loss calculation module; The data acquisition module is used to construct realistic attack scenarios in a specific real-time network environment and generate an initial encrypted malicious traffic dataset; The data preprocessing module is used to preprocess the short-stream data and long-stream data in the initial encrypted malicious traffic data respectively; The interaction graph construction module is used to construct a traffic interaction graph based on the processed short-flow data and the processed long-flow data; The first clustering module is used to extract connected components in the traffic interaction graph and cluster the connected components to obtain normal components and abnormal components. The second clustering module is used to search and cluster the anomalous components to obtain suspicious anomalous components; The loss calculation module is used for calculating a loss function in the process of obtaining suspicious abnormal components, and when the loss function is greater than a set threshold T, the suspicious abnormal component is determined as encrypted malicious traffic.
[0018] Compared with the prior art, the present application has the following advantages: 1. The present application solves the problems of the prior art that unknown encrypted malicious traffic cannot be detected and prior knowledge of known attacks is relied on, by constructing a real attack scene to generate a data set covering three types of malicious traffic, using a differential preprocessing strategy of short stream aggregation and long stream histogram feature extraction, combining traffic interaction graph modeling and connected component analysis, and mining potential malicious patterns from encrypted traffic metadata, without decrypting the encrypted content, unknown encrypted malicious traffic can be effectively identified, and the detection coverage and scene adaptability are greatly improved.
[0019] 2. The present application solves the limitations of the prior detection method that real-time performance is poor and clustering easily introduces irrelevant edges, by extracting connected components through depth-first search, using dual OPTICS clustering of connected component clustering and abnormal component edge pre-clustering, combining efficient local search of the KD-Tree algorithm, realizing fast exclusion of normal traffic and precise focusing of abnormal traffic, and reducing the misjudgment rate by quantifying the malicious degree through a multi-dimensional loss function, meeting the detection efficiency requirements of real-time network environment.
[0020] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS
[0021] Figure 1 is a flow chart of a detection method for unknown encrypted malicious traffic provided by an embodiment of the present application; Figure 2 is a schematic diagram of a detection system for unknown encrypted malicious traffic provided by an embodiment of the present application. DETAILED DESCRIPTION
[0022] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined purpose, the scheme according to the present application will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0023] The foregoing and other technical contents, features and effects of the present application can be clearly presented in the specific embodiment description below with reference to the accompanying drawings. Through the description of the specific embodiments, the technical means and effects adopted by the present application to achieve the predetermined purpose can be understood more deeply and specifically. However, the accompanying drawings are provided for reference and illustration only, and are not used to limit the technical scheme of the present application.
[0024] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one from another entity or action without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process or method that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process or method. Elements defined by an expression "comprising a" do not exclude the presence of additional identical elements in the process or method comprising the defined element.
[0025] As Figure 1 shown, a flow chart of a method for detecting unknown encrypted malicious traffic provided for embodiment 1 of the present application, comprising the following steps: A real attack scene is constructed in a real-time network environment to generate an initial encrypted malicious traffic dataset.
[0026] A specific real-time network is a special network environment built for generating an initial encrypted malicious traffic dataset; the malicious encrypted traffic dataset includes encrypted flood traffic, encrypted malicious network traffic, and encrypted traffic generated by malware; wherein the encrypted flood traffic is generated by initiating repeated attack behaviors, specifically including using low-rate TCP attacks, Crossfire attacks, and encrypted attacks initiated by exploiting protocol vulnerabilities; the encrypted malicious network traffic is generated by initiating network attacks using automated attack tools, collecting instances of these network attacks, and using the HTTPS protocol for encryption to hide malicious intent; the encrypted traffic generated by malware is generated by replaying a public malware traffic dataset in a real-time network environment. The dataset in the present application is generated based on a real attack scene, avoiding the problem of artificial data construction being out of touch with the actual network environment.
[0027] The short flow data and long flow data in the initial encrypted malicious traffic dataset are preprocessed respectively.
[0028] Short flow data meeting the triple constraint conditions are screened: short flows have the same source address and / or target address, consistent protocol type, and the number of short flows ≥ 15; the short flow data meeting the constraint conditions are aggregated and their features are retained to obtain short flow aggregation edges and the features of the short flow aggregation edges, i.e., the processed short flow data. The features include feature sequences and flow four-tuples composed of source IP, target IP, source port, and target port.
[0029] Detect whether there is a flow break point in the long flow data, if there is a flow break point, the continuous part in each flow break point is taken as a first type of continuous long flow data segment; if there is no flow break point, the long flow data is directly taken as a second type of continuous long flow data segment; the first type of continuous long flow data segment and the second type of continuous long flow data segment constitute the continuous long flow data segment; a plurality of intervals are divided in a length unit of 10 bytes, the continuous long flow data segment is classified according to the length into the corresponding interval, and a long flow data histogram is obtained, each continuous long flow data segment with a histogram is a long flow feature edge; the total byte number of the long flow data and the flow duration of the long flow data are obtained.
[0030] The long flow feature edge and the feature of the long flow feature edge are obtained according to the long flow data histogram, the total byte number of the long flow data and the flow duration of the long flow data, that is, the processed long flow data.
[0031] By adopting a differentiated preprocessing strategy according to the characteristics that short flow features are similar but numerous, and long flow features are centralized but long in sequence, key behavior features are retained. By short flow aggregation, the data volume is greatly reduced, and the computational burden of subsequent graph construction and analysis can be reduced; by long flow histogram, the feature storage can be simplified while avoiding the loss of key distribution information.
[0032] A traffic interaction graph is constructed based on the processed short flow data and the processed long flow data.
[0033] The source address and / or the target address of the traffic are taken as the vertices of the traffic interaction graph, the processed short flow aggregation edge and the long flow feature edge are taken as the edges of the traffic interaction graph, and the features of the short flow aggregation edge and the features of the long flow feature edge are integrated into the graph structure, so as to obtain the traffic interaction graph.
[0034] By converting unstructured traffic data into structured graph data, the association relationship between the traffic can be intuitively reflected, which is used for subsequent clustering and improves the analysis efficiency.
[0035] Connected components in the traffic interaction graph are extracted, and the connected components are clustered to obtain normal components and abnormal components.
[0036] A depth-first search (DFS) is performed on the traffic interaction graph to analyze the association relationship between the vertices and the edges, so as to obtain the connected components and the feature dimensions of the connected components; wherein the feature dimensions of the connected components include the number of short flow edges, the number of long flows, the number of short flows, the total byte number of long flows and the total byte number of short flows.
[0037] A density space is constructed based on the feature dimensions of the connected components, and the OPTICS clustering algorithm is used to cluster the connected components to obtain normal components and abnormal components. The normal components are non-malicious traffic, the abnormal components are filtered out from the data, and the normal traffic without malicious risk is excluded, so as to reduce the data volume in subsequent processing.
[0038] The suspicious abnormal component is obtained by searching and clustering the abnormal component.
[0039] The features of the short flow aggregation edge in the abnormal component and the features of the long flow feature edge are normalized by using the min-max normalization to obtain normalized features; While the normalized features are quickly searched by the KD-Tree algorithm, clustering is performed based on the OPTICS clustering algorithm to obtain the suspicious abnormal component.
[0040] By combining the KD-Tree algorithm and the OPTICS clustering algorithm, the problem of complex abnormal component data and low clustering efficiency is solved, achieving the dual goals of fast search and accurate clustering, and further focusing on suspicious abnormal components.
[0041] The loss function in the process of obtaining the suspicious abnormal component is calculated, and when the loss function is greater than a set threshold T, the suspicious abnormal component is determined to be encrypted malicious traffic.
[0042] The loss function in the process of obtaining the suspicious abnormal component includes:
[0043]
[0044]
[0045]
[0046] wherein, represents the cluster center difference loss; is a single clustering center, which is the mean representative of all edge features within the cluster; represents the set of all clustering centers, wherein L is the total number of clusters; f(e) is the feature vector of the edge; represents the L2 norm, which is used to calculate the straight-line distance between the feature vector f(e) and the clustering center min represents the minimum value; represents the cluster time span loss; represents the edge cluster set generated by clustering, that is, the same edge cluster containing edge e; TimeRange() represents the time span operator; represents the cluster flow density loss; represents the size of the edge cluster set ; represents the total loss.
[0047] wherein, the cluster center difference loss is the Euclidean distance between the edge feature vector and the nearest clustering center, which is used to measure the feature similarity between the edge and the same edge; the cluster time span loss Quantify the time distribution range of the flow in the edge cluster; cluster flow density loss Quantify the flow density by the logarithm of the edge cluster size; the total loss is constructed by combining the above three losses, quantifying the malicious degree of suspicious components, and avoiding the misjudgment problem of traditional single-dimensional judgment.
[0048] As Figure 2 shown, a detection system for unknown encrypted malicious flow provided by the embodiment of the application includes a data acquisition module, a data preprocessing module, an interaction graph construction module, a first clustering module, a second clustering module and a loss calculation module.
[0049] The data acquisition module is used to construct a real attack scene in a real-time network environment and generate an initial encrypted malicious flow data set.
[0050] The data preprocessing module is used to preprocess short flow data and long flow data in the initial encrypted malicious flow data respectively.
[0051] The interaction graph construction module is used to construct a flow interaction graph based on the processed short flow data and the processed long flow data.
[0052] The first clustering module is used to extract connected components in the flow interaction graph, and cluster the connected components to obtain normal components and abnormal components.
[0053] The second clustering module is used to search and cluster the abnormal components to obtain suspicious abnormal components.
[0054] The loss calculation module is used to calculate the loss function in the process of obtaining the suspicious abnormal components. When the loss function is greater than a set threshold T, the suspicious abnormal components are determined as encrypted malicious flow.
[0055] As shown in Table 1, the performance results of the embodiment of the application and other existing encrypted malicious flow detection are compared. Table 1 Detection performance results
[0056] AUC is the area under the receiver operating characteristic curve, used to measure the ability to distinguish between malicious flow and non-normal flow, with a value range of 0-1. The closer the value is to 1, the stronger the distinguishing ability. F1 value is the harmonic mean of precision and recall, which comprehensively reflects the precision and integrity, with a value range of 0-1. The closer the value is to 1, the better the comprehensive performance. The precision rate represents the proportion of actual malicious flow in the predicted malicious flow, with a value range of 0-1. The closer the value is to 1, the lower the probability of misjudging normal flow as malicious flow.
[0057] It can be seen that, compared with the existing HyperVision, Whisper, DeepLog and GraphSAGE, the application has significantly improved detection performance.
[0058] The unknown encrypted malicious traffic detection method provided by the application comprises the following steps: constructing a real attack scene in a specific real-time network to generate an initial data set containing three types of malicious traffic; preprocessing short flow and long flow data to obtain short flow aggregation edges and long flow feature edges; then constructing a traffic interaction graph, extracting connected components, and using OPTICS clustering to separate normal components and abnormal components; normalizing the feature of the abnormal component, combining KD-Tree and OPTICS to obtain suspicious abnormal components; finally, calculating the total loss function, and if the loss function value exceeds the set threshold, it is determined as encrypted malicious traffic.
[0059] Through short flow aggregation and long flow histogram processing, data dimensionality reduction is realized, and subsequent calculation overhead is reduced; with the help of double OPTICS clustering and KD-Tree fast search, the efficiency of abnormal traffic positioning is improved; by using a multi-dimensional loss function to quantify the malicious degree, the risk of misjudgment is reduced; the performance of the application in various encrypted malicious traffic detection scenarios is better than that of the existing mainstream methods, and the detection accuracy and real-time performance are considered, effectively solving the problems that the existing technology cannot detect unknown encrypted malicious traffic, relies on known attack prior knowledge and has poor real-time performance, and realizing the detection target of mining potential malicious patterns in encrypted traffic without decrypting the encrypted content, providing an efficient and feasible solution for the defense of unknown encrypted malicious traffic in cyberspace.
[0060] The above is a further detailed description of the application in combination with specific preferred embodiments, and the specific implementation of the application cannot be limited to these descriptions. For ordinary skilled persons in the technical field to which the application belongs, without departing from the concept of the application, a number of simple deductions or substitutions can be made, which should be regarded as falling within the protection scope of the application.
Claims
1. A method for detecting unknown encrypted malicious traffic, characterized in that, The method comprises the following steps: constructing a real attack scene in a specific real-time network environment to generate an initial encrypted malicious traffic dataset; preprocessing short flow data and long flow data in the initial encrypted malicious traffic dataset; constructing a traffic interaction graph based on the processed short flow data and the processed long flow data; extracting connected components in the traffic interaction graph, and clustering the connected components to obtain normal components and abnormal components; searching and clustering the abnormal components to obtain suspicious abnormal components; calculating a loss function in the process of obtaining the suspicious abnormal components, and determining that the suspicious abnormal components are encrypted malicious traffic when the loss function is greater than a set threshold T.
2. The method of claim 1, wherein, The malicious encrypted traffic dataset comprises encrypted flood traffic, encrypted malicious network traffic, and encrypted traffic generated by malicious software.
3. The method of claim 1, wherein the method further comprises: The preprocessing of the short flow data in the initial encrypted malicious traffic dataset comprises: aggregating short flow data meeting a constraint condition and retaining features of the short flow data to obtain short flow aggregation edges and features of the short flow aggregation edges, i.e., the processed short flow data.
4. The method of claim 1, wherein, The preprocessing of the long flow data in the initial encrypted malicious traffic dataset comprises: obtaining continuous long flow data segments according to flow breaking points of the long flow data, and classifying the continuous long flow data segments into corresponding intervals according to lengths of the continuous long flow data segments to obtain a long flow data histogram; obtaining long flow feature edges and features of the long flow feature edges according to the long flow data histogram, a total byte number of the long flow data, and a flow duration of the long flow data, i.e., the processed long flow data.
5. The method of claim 1, wherein the method further comprises: The extraction of the connected components in the traffic interaction graph and the clustering of the connected components into the normal components and the abnormal components comprise: processing the traffic interaction graph by using a depth-first search method to obtain connected components and feature dimensions of the connected components; clustering the connected components by using an OPTICS clustering algorithm based on the feature dimensions of the connected components to obtain the normal components and the abnormal components.
6. The method of claim 1, wherein the method further comprises: The searching and clustering of the abnormal components to obtain the suspicious abnormal components comprise: normalizing features of short flow aggregation edges and features of long flow feature edges in the abnormal components by using minimum-maximum normalization to obtain normalized features; performing fast searching on the normalized features by using a KD-Tree algorithm, and clustering based on the OPTICS clustering algorithm to obtain the suspicious abnormal components.
7. The method of claim 1, wherein the method further comprises: The loss function in the process of obtaining the suspicious abnormal components comprises: where, represents the cluster center difference loss; is the single cluster center, which is the mean representation of all edge features within the cluster; represents the set of all cluster centers, where L is the total number of clusters; f(e) is the feature vector of edge e; represents the L2 norm, which is used to calculate the straight-line distance between the feature vector f(e) and the cluster center min represents the minimum value; represents the cluster time span loss; represents the edge cluster set generated by clustering, i.e., the same edge cluster containing edge e; TimeRange() represents the time span operator; represents the cluster flow density loss; represents the size of the edge cluster set ; represents the total loss.
8. A system for detecting unknown encrypted malicious traffic, the system comprising: performing the detection method of unknown encrypted malicious traffic in any one of claims 1 to 7, which comprises a data acquisition module, a data preprocessing module, an interaction graph construction module, a first clustering module, a second clustering module, and a loss calculation module; the data acquisition module is configured to construct a real attack scene in a specific real-time network environment to generate an initial encrypted malicious traffic dataset; the data preprocessing module is configured to preprocess short flow data and long flow data in the initial encrypted malicious traffic dataset; the interaction graph construction module is configured to construct a traffic interaction graph based on the processed short flow data and the processed long flow data; the first clustering module is configured to extract connected components in the traffic interaction graph, and cluster the connected components to obtain normal components and abnormal components; the second clustering module is configured to search and cluster the abnormal components to obtain suspicious abnormal components; and the loss calculation module is configured to calculate a loss function in the process of obtaining the suspicious abnormal components, and determine that the suspicious abnormal components are encrypted malicious traffic when the loss function is greater than a set threshold T. The second clustering module is configured to search and cluster the abnormal components to obtain suspicious abnormal components. The loss calculation module is configured to calculate a loss function in the process of obtaining the suspicious abnormal components, and determine that the suspicious abnormal components are the encrypted malicious traffic when the loss function is greater than a set threshold T.