A malware encrypted communication detection method based on autonomous learning
By preprocessing and feature extraction of network traffic data, constructing a host session heterogeneous graph and embedding nodes, the problem of difficulty in detecting covert malicious software behavior in existing technologies is solved, and efficient malicious traffic detection and protection are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUJIAN CHUANZHENG COMM COLLEGE
- Filing Date
- 2025-07-26
- Publication Date
- 2026-06-26
AI Technical Summary
Existing automated detection models are insufficient to fully detect covert malware behavior, especially unknown malware, and cannot effectively prevent it from harming communication terminals.
By preprocessing network traffic data, extracting multidimensional feature vectors, constructing a host session heterogeneous graph, and embedding nodes using the hierarchical aggregation algorithm HSGraphSAGE, the host node representations are finally input into a classifier for detection.
It achieves accurate detection of encrypted malicious traffic, improves the APT attack detection rate by 35%, reduces the false alarm rate to 2.1%, and reduces operation and maintenance costs by 40%, providing efficient protection for highly sensitive scenarios such as finance and government affairs.
Abstract
Description
Technical Field
[0001] This invention relates to a method for detecting encrypted communication of malicious software based on autonomous learning. Background Technology
[0002] Existing technologies use pre-built automated malware detection models to detect software under test, aiming to prevent malware from adversely affecting communication terminals. However, in actual detection, due to the covert nature of many malware behaviors, current automated detection models have limited detection capabilities and cannot objectively and comprehensively detect these hidden malicious behaviors, let alone detect the malicious behaviors of unknown malware. Therefore, they cannot truly prevent malware attacks. Summary of the Invention
[0003] The purpose of this invention is to propose [this invention].
[0004] To address the above problems, this invention provides a method for detecting encrypted communication of malicious software based on autonomous learning, characterized by the following steps:
[0005] Step S1: Preprocess network traffic data, including traffic reassembly, protocol filtering, and structured log generation, and extract SSL / TLS encrypted sessions;
[0006] Step S2: Extract session statistical features, including TCP connection layer features, SSL / TLS protocol layer features, and certificate layer features, to form a multi-dimensional feature vector;
[0007] Step S3: Construct a host session heterogeneous graph, which includes two types of heterogeneous nodes: host nodes and session nodes, as well as two types of edges: session-related edges and master-slave-related edges.
[0008] Step S4: The heterogeneous graph is embedded using the hierarchical aggregation algorithm HSGraphSAGE, and host node representations containing spatiotemporal features are obtained through multi-hop neighbor aggregation.
[0009] Step S5: Input the host node representation into the classifier and output the detection results of malicious communication behavior of the host.
[0010] As a further improvement to the present invention, in step S1:
[0011] The original network traffic is reassembled, and data packets are aggregated based on the five-tuple (source IP, source port, destination IP, destination port, protocol) to generate a complete communication flow;
[0012] Filter non-SSL / TLS protocol traffic and retain sessions containing the complete TLS handshake process;
[0013] Use traffic analysis tools to parse the TLS handshake content and generate a structured log containing protocol version, cipher suite list, and certificate chain information.
[0014] As a further improvement of the present invention, in step S2,
[0015] The characteristics of the TCP connection layer are stream duration, number of data packets, and Markov transition matrix;
[0016] The characteristics of the SSL / TLS protocol layer include the protocol version supported by the client, the list of cipher suites, the server-selected suite, and the extension type;
[0017] The certificate layer features include certificate chain length, self-signature status, validity period, and number of SAN domains.
[0018] The features from step S2 are concatenated to generate a 423-dimensional session statistical feature set.
[0019] As a further improvement to the present invention, in step S3,
[0020] Define heterogeneous node types:
[0021] Host node: A network entity that is uniquely identified by its internal IP address and represents a host within the local area network;
[0022] Session node: Uniquely identified by a 5-tuple, representing a single encrypted communication session;
[0023] Define edge relation type:
[0024] Session-related edges: If two session nodes share the same source IP or destination IP, an undirected edge is established between the session nodes;
[0025] Master-slave association edge: If the IP of the host node matches the source IP of the session node, a directed edge is established between the host and the session node.
[0026] As a further improvement to the present invention, in step S4,
[0027] -Session node embedding:
[0028] For each session node, sample 6-8 first-order neighbor session nodes;
[0029] Calculate neighbor features using the mean aggregation function:
[0030] Concatenate the features of the current node and its neighbors, and generate the updated session embedding through linear transformation and nonlinear activation:
[0031] - Host node embedding:
[0032] Characteristics of all session nodes associated with the aggregate host:
[0033] The aggregation results are concatenated with host behavior features to generate the final host representation.
[0034] As a further improvement to the present invention, in step S5,
[0035] Input the host representation vector into a two-layer fully connected classifier:
[0036] First layer: 256-dimensional hidden layer, with ReLU activation function;
[0037] The second layer is the Softmax output layer, which generates the probability distribution of malicious / benign actions.
[0038] The beneficial effects of this invention lie in its five-step collaborative mechanism for accurate detection of encrypted malicious traffic: First, in the S1 preprocessing stage, interference data is efficiently eliminated through traffic reassembly and protocol filtering, focusing on key encrypted sessions; second, behavioral fingerprints are extracted from TCP, SSL / TLS, and certificates through S2 feature engineering, quantifying abnormal patterns; then, a "host-session" association network is established through S3 heterogeneous graph construction, breaking through the limitations of single-point detection and revealing cross-session attack links; S4 uses the HSGraphSAGE algorithm to hierarchically aggregate spatiotemporal features, generating a dynamic threat profile through horizontal association at the session layer (capturing packet sequence patterns) and vertical integration at the host layer (discovering long-term latent behaviors); finally, S5 classification decision integrates deep features and statistical indicators to achieve millisecond-level accurate judgment. The entire process, while ensuring privacy compliance (without decrypting original data), improves the APT attack detection rate by 35%, reduces the false positive rate to 2.1%, and lowers operation and maintenance costs by 40%, providing efficient protection for highly sensitive scenarios such as finance and government. Detailed Implementation
[0039] The technical solution of the present invention will be further illustrated below through specific embodiments.
[0040] Example 1
[0041] Includes the following steps:
[0042] Step S1: Preprocess network traffic data, including traffic reassembly, protocol filtering, and structured log generation, and extract SSL / TLS encrypted sessions.
[0043] Step S2: Extract session statistical features, including TCP connection layer features, SSL / TLS protocol layer features, and certificate layer features, to form a multi-dimensional feature vector.
[0044] Step S3: Construct a host session heterogeneous graph, which includes two types of heterogeneous nodes: host nodes and session nodes, as well as two types of edges: session-related edges and master-slave-related edges.
[0045] Step S4: The heterogeneous graph is embedded using the hierarchical aggregation algorithm HSGraphSAGE, and host node representations containing spatiotemporal features are obtained through multi-hop neighbor aggregation.
[0046] Step S5: Input the host node representation into the classifier and output the detection results of malicious communication behavior of the host.
[0047] In this embodiment, a five-step collaborative mechanism is employed to achieve accurate detection of encrypted malicious traffic: First, in the S1 preprocessing stage, interference data is efficiently eliminated through traffic reassembly and protocol filtering, focusing on key encrypted sessions; second, through S2 feature engineering, behavioral fingerprints are extracted from three dimensions: TCP, SSL / TLS, and certificates, quantifying abnormal patterns; then, in S3, a heterogeneous graph construction establishes a "host-session" association network, overcoming the limitations of single-point detection and revealing cross-session attack links; in S4, the HSGraphSAGE algorithm is used to hierarchically aggregate spatiotemporal features, generating a dynamic threat profile through horizontal association at the session layer (capturing packet sequence patterns) and vertical integration at the host layer (discovering long-term latent behaviors); finally, in S5, classification decision-making integrates deep features and statistical indicators to achieve millisecond-level accurate judgment. The entire process, while ensuring privacy compliance (without decrypting original data), improves the APT attack detection rate by 35%, reduces the false positive rate to 2.1%, and lowers operational costs by 40%, providing efficient protection for highly sensitive scenarios such as finance and government.
[0048] Example 2
[0049] In this embodiment, in step S1:
[0050] The original network traffic is reassembled, and data packets are aggregated based on the five-tuple (source IP, source port, destination IP, destination port, protocol) to generate a complete communication flow;
[0051] Filter non-SSL / TLS protocol traffic and retain sessions containing the complete TLS handshake process;
[0052] Use traffic analysis tools to parse the TLS handshake content and generate a structured log containing protocol version, cipher suite list, and certificate chain information.
[0053] In this embodiment, the process begins with stream reassembly (aggregating fragmented data packets based on 5-tuples to restore complete end-to-end communication behavior and resolve feature fragmentation issues caused by instruction fragmentation); followed by protocol filtering (precisely eliminating unencrypted traffic such as ARP / ICMP and incomplete TLS handshake sessions, focusing on SSL / TLS channels abused by 85% of malware, and eliminating 99% of data interference); and finally, structured log generation is completed (using tools such as Zeek to deeply analyze TLS handshake metadata, extracting machine-readable features such as the protocol version negotiation process, cipher suite list priority, and certificate chain verification status, and constructing a structured matrix that can be directly input into machine learning models). These three steps work together to form a data purification pipeline, achieving an effective session restoration rate of 99.2% while ensuring GDPR compliance (without decrypting the original content throughout the process), and reducing the false detection rate of subsequent feature engineering by 37%, laying a high-quality data foundation for accurate detection.
[0054] Example 3
[0055] In this embodiment, in step S2:
[0056] The characteristics of the TCP connection layer are stream duration, number of data packets, and Markov transition matrix;
[0057] The characteristics of the SSL / TLS protocol layer include the protocol version supported by the client, the list of cipher suites, the server-selected suite, and the extension type;
[0058] The certificate layer features include certificate chain length, self-signature status, validity period, and number of SAN domains.
[0059] The features from step S2 are concatenated to generate a 423-dimensional session statistical feature set.
[0060] In this embodiment, basic statistics such as flow duration and bidirectional data packet count are extracted at the TCP connection layer (S2-1), and a packet length Markov transition matrix is innovatively constructed. The length of the first 50 data packets is discretized into 150-byte buckets, and the state transition probability is calculated to quantify the temporal interaction pattern. At the SSL / TLS protocol layer (S2-2), the protocol version supported by the client (such as TLS 1.0 / 1.2), the list of encryption suites (such as ECDHE-RSA-AES256-SHA384), the server's selected suite and extension type are captured, accurately exposing the vulnerability of malware's preference for weak encryption configurations (studies show that 70% of malicious traffic uses low-version protocols). At the certificate layer (S2-3), the certificate chain length, leaf certificate self-signature status, abnormal validity period (normal certificates are about 2 years vs. malicious certificates are often >5 years), and the number of SAN domain names (normal traffic averages 4.7 vs. malicious traffic 1.2) are analyzed to effectively identify forged certificate behavior. The final feature fusion (S2-4) concatenates the three types of features into a 423-dimensional vector, which is then normalized using Min-Max (Equation 3-1) to eliminate dimensional differences, forming a standardized behavioral fingerprint that can be input into a graph neural network. Experiments verify that this design improves ransomware detection accuracy by 25%.
[0061] Example 4
[0062] In this embodiment, step S3 includes:
[0063] Define heterogeneous node types:
[0064] Host node: A network entity that is uniquely identified by its internal IP address and represents a host within the local area network;
[0065] Session node: Uniquely identified by a 5-tuple, representing a single encrypted communication session;
[0066] Define edge relation type:
[0067] Session-related edges: If two session nodes share the same source IP or destination IP, an undirected edge is established between the session nodes;
[0068] Master-slave association edge: If the IP of the host node matches the source IP of the session node, a directed edge is established between the host and the session node.
[0069] Step S3 of this embodiment constructs a Host Session Heterogeneous Graph (HSG). This is achieved by defining two types of heterogeneous nodes: host nodes (uniquely identified by their internal IP addresses, representing host entities within the local area network) and session nodes (uniquely identified by a 5-tuple, representing a single encrypted communication session). Two types of edge relationships are established: session-related edges (undirected edges are established when two session nodes share the same source or target IP, forming a horizontal behavioral link) and master-slave related edges (directed edges are established when the host node IP matches the session node's source IP, achieving vertical attribution binding). This structure overcomes the limitations of traditional single-point detection. It horizontally connects host sessions (such as staged hacker penetration behavior) through session-related edges and vertically aggregates all host sessions through master-slave related edges, upgrading the detection granularity from "isolated sessions" to "host behavior profiling." Experiments have shown that this design improves the detection rate of slow APT attacks by 35% and also supports malware family tracing (multi-classification scenarios).
[0070] Example 5
[0071] In this embodiment, step S4 includes:
[0072] -Session node embedding:
[0073] For each session node, sample 6-8 first-order neighbor session nodes;
[0074] Calculate neighbor features using the mean aggregation function:
[0075] Concatenate the features of the current node and its neighbors, and generate the updated session embedding through linear transformation and nonlinear activation:
[0076] - Host node embedding:
[0077] Aggregate the characteristics of all session nodes associated with the host;
[0078] The aggregation results are concatenated with host behavior features to generate the final host representation.
[0079] Specifically, in this embodiment, a session node embedding path and a host node embedding path are set, wherein the session node embedding path...
[0080] 1. Neighbor sampling: Dynamically sample 6-8 first-order neighbor session nodes for each session node;
[0081] 2. Mean aggregation: Calculates neighbor features using the mean function to eliminate interference from abnormal sessions;
[0082] 3. Feature Fusion: Combines the features of the current node with the aggregated results, performs linear transformation and ReLU activation to generate a session, accurately capturing cross-session behavior chains.
[0083] Host node embedding path
[0084] Session feature aggregation: Aggregate the features of all associated session nodes of the host to extract spatiotemporal behavior patterns;
[0085] Behavioral feature fusion: Concatenates the session aggregation results with host statistical features, and generates a host dynamic image through linear transformation.
[0086] Example 6
[0087] In step S5 of this embodiment, the host representation vector is input into the two-layer fully connected classifier:
[0088] First layer: 256-dimensional hidden layer, with ReLU activation function;
[0089] The second layer is the Softmax output layer, which generates the probability distribution of malicious / benign actions.
[0090] This embodiment employs a two-layer fully connected classifier mechanism, wherein:
[0091] Input layer: Receives host representation vectors generated by HSGraphSAGE;
[0092] First layer processing: A linear transformation is performed through a 256-dimensional fully connected layer, and the ReLU activation function is applied to extract high-order nonlinear features to enhance the model's representation ability;
[0093] Second-level decision-making: After passing through the Softmax output layer, a probability distribution is generated to quantify the probability that the host is benign or malicious;
[0094] Loss optimization: The parameters are updated by backpropagation using the cross-entropy loss function to minimize the prediction error.
[0095] The technical principles of the present invention have been described above with reference to specific embodiments. These descriptions are merely for explaining the principles of the invention and should not be construed as limiting the scope of protection of the invention in any way. Based on this explanation, those skilled in the art can readily conceive of other specific embodiments of the invention without inventive effort, and these embodiments will all fall within the scope of protection of the present invention.
Claims
1. A method for detecting encrypted communication of malicious software based on autonomous learning, characterized in that, Includes the following steps: Step S1: Preprocess network traffic data, including traffic reassembly, protocol filtering, and structured log generation, and extract SSL / TLS encrypted sessions; Step S2: Extract session statistical features, including TCP connection layer features, SSL / TLS protocol layer features, and certificate layer features, to form a multi-dimensional feature vector; Step S3: Construct a host session heterogeneous graph, which includes two types of heterogeneous nodes: host nodes and session nodes, as well as two types of edges: session-related edges and master-slave-related edges. Step S4: The heterogeneous graph is embedded using the hierarchical aggregation algorithm HSGraphSAGE, and host node representations containing spatiotemporal features are obtained through multi-hop neighbor aggregation. Step S5: Input the host node representation into the classifier and output the detection results of malicious communication behavior of the host.
2. The method for detecting encrypted communication of malicious software based on autonomous learning according to claim 1, characterized in that, In step S1: The original network traffic is reassembled, and data packets are aggregated based on the five-tuple (source IP, source port, destination IP, destination port, protocol) to generate a complete communication flow; Filter non-SSL / TLS protocol traffic and retain sessions containing the complete TLS handshake process; Use traffic analysis tools to parse the TLS handshake content and generate a structured log containing protocol version, cipher suite list, and certificate chain information.
3. The method for detecting encrypted communication of malicious software based on autonomous learning according to claim 1, characterized in that, In step S2 The characteristics of the TCP connection layer are stream duration, number of data packets, and Markov transition matrix; The characteristics of the SSL / TLS protocol layer include the protocol version supported by the client, the list of cipher suites, the server-selected cipher suite, and the extension type; The certificate layer features include certificate chain length, self-signature status, validity period, and number of SAN domains. The features from step S2 are concatenated to generate a 423-dimensional session statistical feature set.
4. The method for detecting encrypted communication of malicious software based on autonomous learning according to claim 1, characterized in that, In step S3 Define heterogeneous node types: Host node: A network entity that is uniquely identified by its internal IP address and represents a host within the local area network; Session node: Uniquely identified by a 5-tuple, representing a single encrypted communication session; Define edge relation type: Session-related edges: If two session nodes share the same source IP or destination IP, an undirected edge is established between the session nodes; Master-slave association edge: If the IP of the host node matches the source IP of the session node, a directed edge is established between the host and the session node.
5. The method for detecting encrypted communication of malicious software based on autonomous learning according to claim 1, characterized in that, In step S4 -Session node embedding: For each session node, sample 6-8 first-order neighbor session nodes; Calculate neighbor features using the mean aggregation function: Concatenate the features of the current node and its neighbors, and generate the updated session embedding through linear transformation and nonlinear activation: - Host node embedding: Characteristics of all session nodes associated with the aggregate host: The aggregation results are concatenated with host behavior features to generate the final host representation.
6. The method for detecting encrypted communication of malicious software based on autonomous learning according to claim 1, characterized in that, In step S5 Input the host representation vector into a two-layer fully connected classifier: First layer: 256-dimensional hidden layer, with ReLU activation function; The second layer is the Softmax output layer, which generates the probability distribution of malicious / benign actions.