Encrypted traffic malicious behavior identification method and system

Abstract

The application discloses an encrypted traffic malicious behavior identification method and system, and belongs to the technical field of network security, and the method comprises the following steps: a metadata extraction module extracts TLS handshake features, certificate chain data and traffic statistical sequences from encrypted traffic; a feature encoding module converts variable-length sequences into fixed-dimension vectors by adopting position encoding and a multi-head attention mechanism; a deep classification module outputs behavior classification probabilities through a plurality of self-attention encoding layers; a certificate reputation evaluation module calculates a certificate reputation score based on multiple dimensions; and a collaborative decision module fuses the outputs of the modules to generate a session risk score and an identification result, so that the application realizes malicious behavior identification without decrypting the traffic.

Description

Methods and Systems for Identifying Malicious Behavior in Encrypted Traffic Technical Field

[0001] This invention relates to the field of network security technology, specifically to a method and system for identifying malicious behavior in encrypted traffic. Background Technology

[0002] With increasing awareness of cybersecurity and growing demand for privacy protection, TLS / SSL encrypted communication protocols have been widely deployed on the internet. Statistics show that encrypted traffic now accounts for over 90% of global network traffic, providing effective security protection for users' online activities and communications. However, this privacy protection mechanism is also being exploited by malicious attackers. An increasing number of malware programs are using encrypted channels to conceal their malicious activities and evade monitoring by traditional security detection systems. Research indicates that over 85% of malware has begun using encrypted communication channels for command and control communications and data leakage.

[0003] Existing technologies for detecting encrypted malicious traffic mainly fall into two categories: deep packet inspection (DPI) and machine learning-based detection. DPI examines the packet payload by decrypting encrypted traffic, but this method faces significant computational overhead and risks user privacy violations, making it difficult to deploy in large-scale network environments. Machine learning-based detection technologies classify encrypted traffic by extracting statistical features, but existing methods suffer from shortcomings in the comprehensiveness of feature extraction and the effectiveness of feature representation.

[0004] Chinese invention CN115589314A discloses a method for detecting and identifying encrypted malicious traffic based on deep learning. This method collects network traffic in real time through an IoT threat sensing terminal and uses a malicious encrypted traffic detection model and a malicious encrypted traffic attack identification model to identify the network traffic. The malicious encrypted traffic detection model uses a single classifier to train on a normal dataset to identify abnormal traffic, and then uses binary classification training to achieve malicious traffic detection. The malicious encrypted traffic attack identification model extracts the time-series features of data packets to establish a malicious encrypted traffic attack fingerprint database, and uses a GRU-DNN neural network model for training and identification. This technical solution can achieve a certain degree of detection of encrypted malicious traffic and identification of attack types.

[0005] However, the above-mentioned technical solutions have the following technical problems: First, the feature extraction dimension is singular, focusing only on the time sequence features of data packets and failing to fully utilize the protocol features and certificate chain attribute information of the TLS handshake phase, resulting in insufficient ability to identify new covert attacks; Second, the feature encoding method is fixed, using a fixed-dimensional feature matrix representation, which makes it difficult to handle the encoding problem of variable-length sequence features in real network environments, limiting the flexibility and expressive power of feature representation; Third, there is a lack of a certificate reputation assessment mechanism, making it impossible to effectively identify and mark high-risk certificates such as self-signed certificates and short-validity certificates; Fourth, the detection models are independent of each other, lacking a collaborative decision-making mechanism, making it difficult to achieve effective fusion of multi-dimensional features and comprehensive risk assessment. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and system for identifying malicious behavior in encrypted traffic. This method can accurately identify malicious behavior in encrypted traffic without decrypting the traffic content, through multi-dimensional metadata feature extraction, variable-length sequence adaptive encoding, certificate reputation assessment, and collaborative decision fusion.

[0007] The first aspect of the present invention provides a method for identifying malicious behavior in encrypted traffic, comprising: a metadata extraction module acquiring encrypted network traffic data, and extracting handshake message feature data from the TLS handshake phase, certificate chain data from the certificate exchange phase, and traffic statistics sequence from the session transmission phase without decrypting the traffic content; wherein the handshake message feature data includes a ClientHello extended field sequence, a cipher suite list, and a set of supported version numbers; the certificate chain data includes certificate validity information, issuing authority identifier, and a list of alternative subject names; and the traffic statistics sequence includes a packet size sequence, packet interval time sequence, and uplink and downlink byte statistics; a feature encoding module receiving the handshake message feature data, certificate chain data, and traffic statistics sequence, assigning position information to each element in the variable-length sequence using position encoding, calculating the association weights between sequence elements using a multi-head attention mechanism, and performing weighted aggregation of sequence elements based on the association weights to generate a TLS handshake feature vector, a certificate attribute feature vector, and a traffic sequence. The system comprises three modules: a statistical feature vector, a deep classification module, and a traffic statistics feature vector. The deep classification module receives TLS handshake feature vectors, certificate attribute feature vectors, and traffic statistics feature vectors. It constructs a fused input vector by concatenating these features and inputs it into a classification network containing multiple self-attention encoding layers. The output is a behavior classification probability vector for a preset malicious behavior category. The certificate reputation assessment module receives certificate chain data and calculates certificate anomaly metrics based on three dimensions: certificate validity period, issuing authority credibility, and number of alternative subject names. It generates a certificate reputation score based on these anomaly metrics and generates risk labeling information for certificates that meet preset risk conditions. The collaborative decision-making module receives the behavior classification probability vector, certificate reputation score, and risk labeling information. It determines the behavior type based on the comparison between the maximum probability value in the behavior classification probability vector and a preset credibility threshold. It then performs a weighted fusion calculation of the certificate reputation score and the behavior type determination to calculate a session risk score, outputting the session risk score, the abnormal behavior type determination result, and a list of suspicious certificate fingerprints.

[0008] Preferably, the extraction of handshake message feature data includes: extracting extended field type identifier sequences and corresponding extended field length sequences from the ClientHello message, extracting suite identifier sequences from the cipher suite list, and extracting protocol version identifier sequences from the supported version number set.

[0009] Preferably, the extraction of the traffic statistics sequence includes: statistically analyzing the distribution of data packet size in units of a preset time window, calculating the arrival time interval of adjacent data packets to form a packet interval time sequence, calculating the ratio of uplink bytes to downlink bytes, and identifying burst transmission pattern sequences formed by continuous high-volume transmission.

[0010] Preferably, the certificate reputation assessment module generates risk label information for certificates that meet at least one of the following preset risk conditions: the certificate is a self-signed certificate, the certificate validity period is less than a preset validity period threshold, and the certificate issuing authority is not in the preset trusted issuing authority list.

[0011] A second aspect of the present invention provides a system for identifying malicious behavior in encrypted traffic, comprising: a metadata extraction module for acquiring encrypted network traffic data, extracting handshake message feature data from the TLS handshake phase, extracting certificate chain data from the certificate exchange phase, and extracting traffic statistics sequences from the session transmission phase without decrypting the traffic content; a feature encoding module for receiving the handshake message feature data, certificate chain data, and traffic statistics sequences, converting the variable-length sequences into fixed-dimensional vectors using positional encoding and multi-head attention mechanisms, and generating TLS handshake feature vectors, certificate attribute feature vectors, and traffic statistics feature vectors; a deep classification module for receiving the TLS handshake feature vectors, certificate attribute feature vectors, and traffic statistics feature vectors, and outputting behavior classification probability vectors for preset malicious behavior categories through a classification network containing multiple layers of self-attention encoding; a certificate reputation evaluation module for receiving certificate chain data, calculating a certificate reputation score based on the certificate validity period, issuing authority credibility, and number of alternative subject names, and generating risk label information for certificates that meet preset risk conditions; and a collaborative decision-making module for receiving the behavior classification probability vectors, certificate reputation scores, and risk label information, calculating a session risk score based on weighted fusion, and outputting a session risk score, an abnormal behavior type determination result, and a list of suspicious certificate fingerprints.

[0012] The beneficial effects of this invention are as follows: First, by using a multi-dimensional metadata extraction mechanism, side-channel features are comprehensively extracted from the TLS handshake phase, certificate exchange phase, and session transmission phase, significantly improving the completeness and discriminative power of the features; Second, by employing a feature encoding method that combines positional encoding with a multi-head attention mechanism, the problem of encoding variable-length sequence features is effectively solved, enhancing the flexibility and expressive power of feature representation; Third, a certificate reputation assessment mechanism is established, which can identify and mark the risks of self-signed certificates, short-term certificates, and certificates from abnormal issuing authorities; Fourth, by using a collaborative decision-making module, multi-dimensional features are effectively fused, providing comprehensive risk assessment capabilities and outputting comprehensive identification results, including session risk scores, abnormal behavior type determination, and a list of suspicious certificates. Attached Figure Description

[0013] Figure 1 is a flowchart of the method for identifying malicious behavior in encrypted traffic according to the present invention.

[0014] Figure 2 is an architecture diagram of the encrypted traffic malicious behavior identification system of the present invention. Detailed Implementation

[0015] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0016] Figure 1 shows a flowchart of the encrypted traffic malicious behavior identification method of the present invention. In one embodiment of the present invention, the method mainly includes five core processing stages: metadata extraction, feature encoding, deep classification, certificate reputation assessment, and collaborative decision-making. The stages form a closed-loop collaborative architecture for data flow.

[0017] Step S1, Metadata Extraction Stage: The metadata extraction module 1 is responsible for extracting multi-dimensional side-channel feature information from encrypted network traffic, obtaining feature data for subsequent analysis without decrypting the traffic content. In a preferred embodiment of the present invention, the metadata extraction module 1 adopts a passive traffic capture method, acquiring encrypted network traffic data through a traffic mirroring port deployed at the network boundary.

[0018] The metadata extraction module 1's processing flow is divided into three parallel feature extraction branches. The first branch is responsible for extracting TLS handshake message features. When a TLS handshake message after TCP connection establishment is detected, the metadata extraction module 1 first locates the start position of the ClientHello message, and then parses the message header fields and extension fields sequentially. During the parsing of the ClientHello message, the metadata extraction module 1 extracts the following feature data: extension field type identifier sequence, including the type identifier values ​​of fields such as Server Name Indicator (SNI) extension, Supported Elliptic Curve Group (ECG) extension, Signature Algorithm extension, and Application Layer Protocol Negotiation (ALPN) extension; extension field length sequence, recording the byte length of each extension field; cipher suite list, extracting the sequence of cipher suite identifiers supported by the client, each identifier being a two-byte numerical encoding; and supported version number set, extracting the TLS protocol version identifiers supported by the client. In a preferred embodiment of the present invention, the numerical range of the extension field type identifier is 0 to 65535, and the numerical range of the cipher suite identifier follows the IANA TLS cipher suite registration specification.

[0019] It's important to note that different applications and malware often employ different configuration strategies when implementing TLS clients. This leads to significant differences in the order of extended fields, cipher suite priorities, and supported version combinations in the ClientHello message. Legitimate browsers typically support a more complete set of extended fields and newer cipher suites, while some malware may support only limited extended fields and outdated cipher suites to reduce implementation complexity. This variability provides exploitable side-channel information for identifying malicious traffic.

[0020] The second branch is responsible for extracting certificate chain data. When the TLS handshake enters the server certificate exchange phase, metadata extraction module 1 captures the Certificate message sent by the server and parses the certificate chain. The extracted certificate chain data includes: certificate validity information, extracting the effective and expiration times of each certificate and calculating the certificate validity period; issuing authority identifier, extracting the Distinguished Name (DN) field of the certificate issuer; subject alternative name list, extracting the list of domain names contained in the SAN extension of the certificate subject alternative name; and certificate fingerprint, calculating the SHA-256 hash value of the certificate as a unique identifier. In a preferred embodiment of the present invention, the unit for calculating the certificate validity period is days, and the statistical range of the number of subject alternative names includes both DNS type and IP address type.

[0021] Certificate chain data is valuable for identifying malicious traffic because malicious attackers often use self-signed certificates or short-term certificates obtained from non-authoritative organizations to quickly deploy attack infrastructure. Self-signed certificates are characterized by the issuer's Domain Name (DN) being the same as the subject's DN; these certificates do not require the authentication process of a Certificate Authority (CA). Short-term certificates typically have a validity period of no more than 90 days, or even just a few days. This short-term certificate usage pattern aligns with the behavior of attackers who frequently change their infrastructure.

[0022] The third branch is responsible for extracting traffic statistics sequences. During the data transmission phase of a TLS session, the metadata extraction module 1 performs statistical analysis on encrypted data packets, extracting the following feature sequences: data packet size sequence, recording the byte length of each TLS record layer data packet to form a size sequence arranged in chronological order; packet interval sequence, calculating the arrival time difference between adjacent data packets to form a time interval sequence; uplink and downlink byte statistics, respectively counting the cumulative number of bytes from the client to the server and from the server to the client, and calculating the uplink and downlink byte ratio; burst transmission pattern sequence, identifying the transmission patterns of multiple consecutive large data packets, recording the start time, duration, and total number of bytes of the burst.

[0023] In a preferred embodiment of the present invention, the statistical range of the data packet size is the TLS record layer payload length, excluding TCP / IP header overhead. The calculation precision of the packet interval timing is at the microsecond level, which can capture fine-grained timing characteristics in high-speed network environments. The identification of burst transmission patterns adopts a sliding window method. When the average size of N consecutive data packets exceeds a preset burst threshold, it is marked as a burst transmission, where the preferred value of N is 5, and the preferred value of the preset burst threshold is 1000 bytes.

[0024] Traffic statistics sequences can reflect the behavioral patterns of encrypted sessions. Normal HTTPS browsing behavior typically exhibits an alternating request-response pattern, with downlink traffic dominating. However, certain malicious behaviors, such as data breaches, are characterized by uplink traffic dominating, while command and control communications may show a periodic pattern of small data packet interactions. These behavioral differences are reflected in the distribution characteristics of traffic statistics sequences, providing important evidence for identifying malicious behavior.

[0025] In a preferred embodiment of the present invention, the metadata extraction module 1 further performs preprocessing operations on the extracted raw feature data to improve the efficiency and accuracy of subsequent processing. For TLS handshake message feature data, the preprocessing operations include: normalizing the extended field type identifier, mapping the original hexadecimal identifier value to a preset identifier space; truncating or padding the cipher suite list, unifying the sequence length to a preset maximum number of suites (in a preferred embodiment, the maximum number of suites is set to 30); zero-padding sequences shorter than the maximum length, and truncating the first 30 suite identifiers from sequences longer than the maximum length.

[0026] For certificate chain data, preprocessing operations include: converting certificate validity periods into a unified timestamp format for easier subsequent numerical calculations; encoding the issuing authority identifier, converting the distinguishable string name into a numerical code; and deduplicating and sorting the subject alternative name list, eliminating duplicate entries and arranging them in lexicographical order. For traffic statistics sequences, preprocessing operations include: bucketing data packet sizes, discretizing continuous size values ​​into a preset number of size intervals; performing a logarithmic transformation on the packet interval time sequence to compress the dynamic range of the time interval; and normalizing uplink and downlink byte statistics to eliminate orders of magnitude differences between different sessions.

[0027] The output data structure of metadata extraction module 1 is designed as follows: TLS handshake message feature data is organized in the form of a multi-dimensional array, with the first dimension being the number of extended fields and the second dimension being the number of feature attributes for each extended field; certificate chain data is organized in the form of structured records, with each record containing four fields: validity period start and end time, issuing authority code, number of alternative subject names, and certificate fingerprint; traffic statistics sequences are organized in the form of a time series matrix, with the number of rows representing the number of statistical time windows and the number of columns representing the number of statistical features within each window. This structured data organization facilitates subsequent vectorization processing by feature encoding module 2.

[0028] Step S2, the feature encoding stage, involves feature encoding module 2 converting the variable-length sequence features output by metadata extraction module 1 into fixed-dimensional vector representations for subsequent processing by the deep learning model. In one embodiment of the invention, feature encoding module 2 employs a sequence encoding method based on positional encoding and multi-head attention mechanisms, which effectively captures the dependencies between sequence elements and generates feature vectors with rich semantic information.

[0029] The positional encoding process is as follows: For the input variable-length sequence, let the sequence length be... Each sequence element is first mapped to an embedding layer. The original representation vector of the dimension. To enable the model to perceive the positional information of sequence elements, feature encoding module 2 generates a corresponding positional encoding vector for each position. This invention adopts the sine and cosine positional encoding method, and the calculation formula for the positional encoding vector is as follows:

[0030] ,

[0031] ,

[0032] in: Encode the position vector at position The first Values ​​for each dimension; Encode the position vector at position The first Values ​​for each dimension; The position index of the sequence element, with a value ranging from 0 to... ; This is a dimension index, with values ​​ranging from 0 to... ; The dimension of the position encoding vector is the same as the dimension of the original representation vector.

[0033] The positional encoding vector is added to the original representation vector to obtain an enhanced representation vector containing positional information. The advantage of this encoding method is that it can generate positional codes for sequences of arbitrary length, and there is a linear relationship between the relative positions of the positional codes, which is beneficial for the model to learn the positional dependencies between sequence elements. In a preferred embodiment of the invention, the embedding dimension... The value of 256 represents a good balance between representational power and computational efficiency.

[0034] The multi-head attention mechanism processes as follows: Feature encoding module 2 takes the position-enhanced sequence representation as input and calculates the association weights between sequence elements through the multi-head attention mechanism. Let the input sequence be represented as a matrix. The multi-head attention mechanism first processes the input through three sets of learnable linear transformations to obtain the query matrix. Key matrix Sum matrix :

[0035] ,

[0036] in: To query the weight matrix of the transformation; This is the weight matrix for the key transformation; The weight matrix for value transformation; The dimensions of the query and key vectors; is the dimension of the value vector.

[0037] Attention weights are calculated using the scaled dot product attention method, and the formula is as follows:

[0038] ,

[0039] in: Calculate the dot product similarity between the query vector and the key vector to obtain... The similarity matrix; The scaling factor is used to prevent the gradient of the softmax function from vanishing due to excessively large dot product values; the softmax function normalizes each row of the similarity matrix to obtain the attention weight matrix; the attention weight matrix and the value matrix... Multiply them to obtain the weighted aggregated output.

[0040] Multi-head attention executes the above computation process in parallel. Each time, a different transformation weight matrix is ​​used, and then... The outputs of each attention head are concatenated and then linearly transformed to obtain the final output:

[0041] ,

[0042] in: For the first The output of each attention head; This is the weight matrix for the output transformation. In a preferred embodiment of the invention, the number of attention heads... The value is 8, and the dimension of each head is... .

[0043] To generate fixed-dimensional feature vectors from variable-length sequences, feature encoding module 2 adds a global average pooling operation after the multi-head attention output to aggregate the sequence dimensions:

[0044] ,

[0045] in: The output is a fixed-dimensional feature vector; The first multi-head attention output sequence A vector at each position; The sequence length is given.

[0046] Feature encoding module 2 performs the above encoding process on the TLS handshake message feature sequence, certificate attribute feature sequence, and traffic statistics feature sequence, respectively, to generate a TLS handshake feature vector. Certificate attribute feature vector and flow statistics feature vector The dimensions of the three feature vectors are all .

[0047] In a preferred embodiment of the present invention, the feature encoding module 2 further employs a hierarchical attention mechanism to encode features of different granularities. Specifically, for TLS handshake message features, fine-grained attention calculation is first performed within the extended fields, and then coarse-grained attention calculation is performed between the extended fields. This hierarchical structure can simultaneously capture local features within fields and global correlations between fields. For traffic statistics sequences, attention weighting is first applied to statistical features within the time window, and then temporal attention calculation is performed between time windows, which can effectively model the temporal evolution pattern of traffic behavior.

[0048] The feature encoding module 2 is pre-trained using a contrastive learning strategy. Specifically, feature sequences sampled from different time segments of the same encrypted session are considered positive sample pairs, while feature sequences from different sessions are considered negative sample pairs. By maximizing the feature vector similarity of positive sample pairs and minimizing the feature vector similarity of negative sample pairs, feature encoding module 2 can learn discriminative feature representations. After pre-training, the parameters of feature encoding module 2 are used as initialization parameters for end-to-end fine-tuning training on labeled data.

[0049] To enhance the robustness of feature representations, feature encoding module 2 employs data augmentation techniques during training. For TLS handshake message features, data augmentation operations include randomly masking field expansion and randomly swapping the order of cipher suites; for traffic statistics sequences, data augmentation operations include adding Gaussian noise, random time offsets, and random sampling of subsequences. These data augmentation operations improve the model's tolerance to input variations and enhance its generalization ability in real-world deployment environments.

[0050] Feature encoding module 2 also implements a multi-scale feature fusion mechanism. When generating the final feature vector, in addition to global average pooling, global max pooling and attention pooling are performed in parallel. The results of these three pooling methods are then concatenated to obtain the fused feature vector. Global max pooling can capture significant feature peaks in the sequence, while attention pooling can adaptively focus on important sequence positions. The combination of these three pooling methods provides richer feature representations.

[0051] Step S3, the deep classification stage, involves the deep classification module 3 learning the differences in metadata features between normal and malicious encrypted communication, and classifying the encrypted sessions according to their behavior. In one embodiment of the invention, the deep classification module 3 employs a classification network architecture based on multi-layer self-attention encoding layers, which can effectively model the interaction relationships between multi-source features.

[0052] The input processing procedure for deep classification module 3 is as follows: First, the three feature vectors output by feature encoding module 2 are concatenated to construct a fused input vector:

[0053] ,

[0054] in: To fuse the input vectors; This represents a vector concatenation operation. The merged input vectors are mapped back through a linear transformation layer. A 3D space is provided for input into subsequent self-attention encoding layers.

[0055] The self-attention encoding layer is structured as follows: Each self-attention encoding layer contains a multi-head self-attention sub-layer, a first normalization sub-layer, a feedforward neural network sub-layer, and a second normalization sub-layer, connected sequentially. The multi-head self-attention sub-layer uses the same multi-head attention mechanism as feature encoding module 2, but its target is the fused feature vector rather than the sequence representation. The feedforward neural network sub-layer contains two linear transformation layers and a GELU activation function layer located between them.

[0056] ,

[0057] in: and The weights and biases for the first linear transformation layer; and The weights and biases for the second linear transformation layer; Let be the hidden layer dimension of the feedforward network, and preferably take the value of . GELU is the activation function for the Gaussian error linear unit.

[0058] Each sublayer is followed by residual connections and layer normalization:

[0059] ,

[0060] in: This indicates a multi-head self-attention sublayer or a feedforward neural network sublayer; residual connections help with the training of deep networks, and layer normalization helps stabilize the training process.

[0061] In a preferred embodiment of the present invention, the deep classification module 3 includes 6 self-attention encoding layers. After multi-layer encoding processing, the output vector is used by the classification head to predict the behavior category:

[0062] ,

[0063] in: This is the output vector of the last coding layer; and For the weights and biases of the classification layer; To preset the number of malicious behavior categories; This is a probability vector for behavior classification, where each component represents the predicted probability of the corresponding behavior category.

[0064] In a preferred embodiment of the present invention, the preset malicious behavior categories include five categories: normal communication, C2 communication, encrypted tunnel abuse, data leakage, and malware update channel. C2 communication refers to communication between malware and a command and control server, typically exhibiting a periodic heartbeat and command response pattern. Encrypted tunnel abuse refers to unauthorized network penetration using encrypted channels. Data leakage refers to the act of transmitting sensitive data to an external server through an encrypted channel. Malware update channel refers to the act of malware downloading update packages or additional modules from a remote server.

[0065] The training of deep classification module 3 uses the cross-entropy loss function for supervised learning. Let the true class labels of the training samples be... The model outputs a behavior classification probability vector as follows: The formula for calculating cross-entropy loss is as follows:

[0066] ,

[0067] in: This represents the cross-entropy loss value. Total number of categories; For indicator functions, when the true class equals The value is 1 if the condition is met, and 0 otherwise. The first output of the model Class probability value.

[0068] Considering that the number of normal traffic samples in the actual dataset is much larger than the number of malicious traffic samples, deep classification module 3 uses class-weighted cross-entropy loss to alleviate the class imbalance problem. The formula for calculating the weighted cross-entropy loss is as follows:

[0069] ,

[0070] in: For the first The weight coefficient of a class is typically set as the reciprocal of the number of samples in that class or the reciprocal of its square root. In a preferred embodiment of the invention, the weight coefficient is calculated using the effective sample count method, specifically using the following formula: ,in For the first The number of samples in each class The smoothing coefficient is preferably set to 0.9999.

[0071] The deep classification module 3 also introduces label smoothing to improve the model's generalization ability. Label smoothing converts hard labels into soft labels, that is, replacing the original true class label of 1 with a soft label. Replace other category labels that originally had a value of 0 with ,in The smoothing coefficient is preferably set to 0.1. Label smoothing prevents the model from becoming overconfident in the training samples and improves its generalization performance on the test set.

[0072] In a preferred embodiment of the present invention, the deep classification module 3 employs gradient accumulation technology to support large-batch training. Due to memory limitations, the batch size of a single forward propagation is limited. By accumulating gradients from multiple forward propagations before performing a parameter update, the effect of large-batch training can be achieved. In a preferred configuration of the present invention, the actual batch size is set to 32, the gradient accumulation steps are set to 8, and the equivalent batch size is 256.

[0073] The optimizer for deep classification module 3 employs the AdamW algorithm, which introduces weight decay regularization on top of the Adam optimizer. The learning rate scheduling uses a cosine annealing strategy, where the learning rate gradually decreases from its initial value to a minimum, and then periodically increases again. In a preferred embodiment of the invention, the initial learning rate is set to... The minimum learning rate is set to The cosine period is set to 10 epochs, and the weight decay coefficient is set to 0.01.

[0074] Step S4, Certificate Reputation Assessment Stage: The certificate reputation assessment module 4 is responsible for assessing the risks of certificates used in encrypted sessions, identifying suspicious certificates, and generating risk tags. In one embodiment of the present invention, the certificate reputation assessment module 4 calculates certificate anomaly metrics from three dimensions: certificate validity period length, issuing authority credibility, and number of alternative subject names.

[0075] The calculation of the certificate validity period anomaly metric is as follows:

[0076] ,

[0077] in: This is a measure of certificate validity period anomalies, ranging from 0 to 1. The certificate validity period is expressed in days. The preset minimum validity period threshold is preferably set to 30 days; The normal validity period is a reference value, with 365 days being the preferred value. When the certificate validity period is lower than the minimum threshold, the validity period anomaly metric is 1, indicating a high degree of anomaly; when the certificate validity period is between the minimum threshold and the normal reference value, the validity period anomaly metric decreases linearly; when the certificate validity period reaches or exceeds the normal reference value, the validity period anomaly metric is 0.

[0078] The credibility metric for the issuing authority is calculated as follows:

[0079] ,

[0080] in: The abnormality metric for the issuing authority is 1 for a self-signed certificate; 0.7 for a certificate whose issuing authority is not in the preset trusted list; and 0 for a certificate whose issuing authority is in the preset trusted list. In a preferred embodiment of the present invention, the preset trusted issuing authority list includes mainstream certificate authorities such as DigiCert, Let's Encrypt, GlobalSign, Comodo, and Sectigo.

[0081] The calculation of the abnormal metric for alternative subject names is as follows:

[0082] ,

[0083] in: Abnormal metrics for alternative subject names; The number of alternative names for the topic; The upper limit of the normal quantity is 10, which is the preferred value; The threshold for the number of anomalies is 100, which is the preferred value. When the certificate does not contain any alternative subject names, the anomaly metric is 0.5; when the number of alternative subject names is within the normal range, the anomaly metric is 0; when the number of alternative subject names is too high, the anomaly metric increases with the number of alternative subject names. Too many alternative subject names may indicate that the certificate has been used for multiple unrelated domain names, posing a risk of abuse.

[0084] The formula for calculating the overall certificate credibility score is as follows:

[0085] ,

[0086] in: This is a certificate credibility score, ranging from 0 to 1. A higher score indicates a higher level of credibility for the certificate. , , Here are the weighting coefficients for each outlier metric, satisfying... In a preferred embodiment of the present invention, , , The credibility of the issuing authority is given the highest weight because self-signed certificates and certificates from non-authoritative institutions are key characteristics of malicious traffic.

[0087] The certificate reputation assessment module 4 generates risk label information based on the following preset risk conditions: When the certificate is self-signed, a SELF_SIGNED risk label is generated; when the certificate's validity period is less than a preset validity period threshold, a SHORT_VALIDITY risk label is generated; when the certificate authority is not in the preset trusted list, an UNTRUSTED_ISSUER risk label is generated. The same certificate may be labeled with multiple risk types, and the risk label information is stored in association with the certificate fingerprint.

[0088] Step S5, the collaborative decision-making stage, involves the collaborative decision-making module 5 fusing the behavior classification results from the deep classification module 3 and the certificate evaluation results from the certificate reputation assessment module 4 to generate a comprehensive recognition output. In one embodiment of the invention, the collaborative decision-making module 5 employs a weighted fusion strategy to calculate a session risk score and outputs a structured recognition result.

[0089] The process for determining behavior type is as follows: Collaborative decision-making module 5 starts from the behavior classification probability vector. Obtain the maximum probability value and their corresponding behavioral categories :

[0090] ,

[0091] in: The first of the behavior classification probability vectors One component; This is the index of the behavior category corresponding to the highest probability. The collaborative decision-making module 5 will... Compared with the preset confidence threshold If a comparison is made, The behavior type determination result is: The corresponding category; if If the behavior type is determined to be uncertain, then the behavior type determination result is uncertain. In a preferred embodiment of the present invention, a pre-set confidence threshold is used. The value is 0.7.

[0092] The calculation process for the session risk score is as follows: When the behavior type is determined to be malicious, the collaborative decision-making module 5 calculates the session risk score:

[0093] ,

[0094] in: A risk score is assigned to the conversation, with a value ranging from 0 to 1. A higher score indicates a greater risk. As behavioral risk weights, For certificate risk weighting, satisfy ; The predicted probability for the category of malicious behavior; The risk component of the certificate is determined by the credit score; a lower score corresponds to a higher risk component. In a preferred embodiment of the invention, , The behavior classification results are given higher weight. When the behavior type is determined to be normal communication, the session risk score only considers the certificate risk component. When the behavior type is determined to be uncertain, due to insufficient confidence in the model's judgment of the session, the collaborative decision-making module 5 employs a prudent assessment strategy to calculate the session risk score. The maximum probability value is multiplied by the uncertainty type weight coefficient to obtain the uncertainty risk component, which is then added to the certificate risk component. The uncertainty type weight coefficient is set to half the weight of the malicious category, and in a preferred embodiment of the invention, it is set to 0.35. This setting reflects the conservative handling principle for low-confidence judgments, neither overestimating the risk nor ignoring potential threats.

[0095] The output of the collaborative decision-making module 5 includes the following three parts: session risk score. The output includes: an overall risk level of the encrypted session; an abnormal behavior type determination result, including specific malicious behavior categories or normal indicators; and a list of suspicious certificate fingerprints, containing the SHA-256 fingerprints of all certificates marked as risky and their risk label types. These three parts of the output constitute a complete profile of malicious behavior in encrypted traffic, providing decision support for cybersecurity operators.

[0096] In a preferred embodiment of the present invention, the collaborative decision-making module 5 also outputs detailed decision interpretation information, including the main factors leading to the current risk score, the contribution analysis of each dimension of features, and the correlation with similar historical cases. The decision interpretation information is presented in the form of visual charts, including feature importance radar charts, risk component pie charts, and time-series risk trend charts. This interpretable output helps security operations personnel understand the basis of the model's judgments and increases their trust in the identification results.

[0097] The collaborative decision-making module 5 implements a multi-level alarm mechanism. Based on the range of session risk scores, alarm levels are divided into four levels: when... When the alarm level is low, only logs are recorded without triggering an alarm; when When the alarm level is medium, a normal alarm is triggered and added to the pending queue; when When the alarm level is high, an emergency alarm is triggered and pushed to security operations personnel; when When the alarm level is critical, an immediate alert is triggered, and preset response strategies such as blocking connections or isolating endpoints are automatically executed. A multi-level alarm mechanism helps security teams allocate resources effectively and prioritize high-risk events.

[0098] The collaborative decision-making module 5 also implements session correlation analysis. For multiple encrypted sessions from the same source IP address or the same target server, the collaborative decision-making module 5 performs correlation analysis on the identification results and calculates a correlation risk score. If multiple sessions from the same source IP are identified as malicious behavior, or if multiple sessions connect to the same risk-marked server, the correlation risk score will increase accordingly. Correlation analysis can discover covert attack behaviors scattered across multiple sessions, improving the detection capability of coordinated attacks and persistent threats.

[0099] Furthermore, the collaborative decision-making module 5 also implements a feedback optimization mechanism. When security operations personnel manually confirm or correct the identification results, the confirmation information is fed back into the training dataset of the deep classification module 3 for incremental model updates. This closed-loop feedback mechanism continuously improves the model's ability to identify new types of attacks. In a preferred embodiment of the invention, the incremental update employs an online learning method, triggering a fine-tuning of model parameters after accumulating a certain number of confirmed samples. The preferred update cycle is set to trigger an update every 1000 confirmed samples.

[0100] In a preferred embodiment of the present invention, the collaborative decision-making module 5 also implements model version management and a rollback mechanism. After each incremental update, the new version of the model is tested for performance on an independent test set. Only when the performance metrics of the new version are not lower than those of the old version is the new version deployed as a production model. Simultaneously, the model files of the three most recent versions are retained, allowing for a quick rollback to a stable version if the new version performs abnormally in the production environment. This version management mechanism ensures the security and controllability of model updates.

[0101] Figure 2 shows the architecture diagram of the encrypted traffic malicious behavior identification system of the present invention. In one embodiment of the present invention, the system includes a metadata extraction module 1, a feature encoding module 2, a deep classification module 3, a certificate reputation assessment module 4, and a collaborative decision-making module 5.

[0102] Metadata extraction module 1 is used to acquire encrypted network traffic data. Without decrypting the traffic content, it extracts handshake message feature data from the TLS handshake phase, certificate chain data from the certificate exchange phase, and traffic statistics sequences from the session transmission phase. As described in the method embodiment, the handshake message feature data includes a ClientHello extended field sequence, a cipher suite list, and a set of supported version numbers. The certificate chain data includes certificate validity information, issuing authority identifier, and a list of alternative subject names. The traffic statistics sequence includes a packet size sequence, packet interval time sequence, and uplink / downlink byte statistics. In a preferred embodiment of the invention, metadata extraction module 1 uses zero-copy technology to directly parse the packet header in kernel space, avoiding data copying overhead between user space and kernel space and improving processing throughput. Metadata extraction module 1 supports parallel processing of traffic from multiple network interfaces. Each network interface is configured with an independent extraction thread, and the extraction results are passed to subsequent modules through a lock-free queue.

[0103] Feature encoding module 2 receives handshake message feature data, certificate chain data, and traffic statistics sequence output by metadata extraction module 1. It uses positional encoding and multi-head attention mechanisms to convert the variable-length sequence into a fixed-dimensional vector. As described in the method embodiment, feature encoding module 2 uses sine and cosine positional encoding to assign positional information to sequence elements, uses scaled dot product attention to calculate the correlation weights between sequence elements, and generates fixed-dimensional TLS handshake feature vectors, certificate attribute feature vectors, and traffic statistics feature vectors through global average pooling. In a preferred embodiment of the invention, the neural network model of feature encoding module 2 is deployed on a GPU device, utilizing the parallel computing capabilities of the GPU to accelerate matrix operations. Feature encoding module 2 supports batch processing mode, assembling feature sequences from multiple sessions into batch tensors and then feeding them into the model for computation at once, fully utilizing the parallel processing capabilities of the GPU.

[0104] The deep classification module 3 receives the TLS handshake feature vector, certificate attribute feature vector, and traffic statistics feature vector output by the feature encoding module 2. It constructs a fused input vector through feature concatenation and inputs it into a classification network containing multiple self-attention encoding layers, outputting a behavior classification probability vector for a preset malicious behavior category. As described in the method embodiment, the deep classification module 3 contains 6 self-attention encoding layers, and the preset malicious behavior categories include normal communication, C2 communication, encrypted tunnel abuse, data leakage, and malware update channels. In a preferred embodiment of the invention, the model file of the deep classification module 3 is stored in ONNX format, supporting cross-platform deployment and inference engine acceleration. The deep classification module 3 supports hot model updates, loading new versions of model parameters without interrupting service.

[0105] The certificate reputation assessment module 4 receives the certificate chain data output by the metadata extraction module 1, calculates the certificate reputation score based on the certificate validity period, the credibility of the issuing authority, and the number of alternative subject names, and generates risk label information for certificates that meet preset risk conditions. As described in the method embodiment, the preset risk conditions include the certificate being a self-signed certificate, the certificate validity period being less than a preset validity period threshold, and the certificate issuing authority not being in the preset trusted issuing authority list. In a preferred embodiment of the present invention, the certificate reputation assessment module 4 maintains a local certificate reputation cache, directly returning the cached result for certificates that have already been assessed to avoid duplicate calculations. The certificate reputation cache adopts an LRU eviction policy, with a preferred cache capacity of 1 million records and an expiration time of 24 hours for cache entries.

[0106] The collaborative decision-making module 5 receives the behavior classification probability vector output by the deep classification module 3, the certificate reputation score output by the certificate reputation assessment module 4, and risk labeling information. Based on a weighted fusion strategy, it calculates a session risk score and outputs the session risk score, abnormal behavior type determination results, and a list of suspicious certificate fingerprints. As described in the method embodiment, the collaborative decision-making module 5 also implements a feedback optimization mechanism, enabling incremental updates to the deep classification module 3 based on manual confirmation information. In a preferred embodiment of the invention, the collaborative decision-making module 5 interfaces with a security information and event management platform, outputting the identification results in a structured log format to a log collection system, supporting subsequent security event correlation analysis and threat intelligence mining.

[0107] In a preferred embodiment of the present invention, the above modules can be deployed and run on network security devices, such as intrusion detection systems, network traffic analysis devices, or the analysis platform of a security operations center. The metadata extraction module 1 acquires the encrypted traffic to be analyzed through a traffic mirroring interface. After processing by the feature encoding module 2 and the deep classification module 3, the identification results output by the collaborative decision-making module 5 can be provided to the security operations platform for visualization and alarm processing via API interface or logs. The system as a whole supports horizontal scaling deployment, and can cope with the processing needs of high-traffic scenarios by deploying multiple computing nodes and adopting load balancing strategies.

[0108] To verify the effectiveness of the technical solution of this invention, the following experimental tests were conducted. The test environment used a server equipped with an NVIDIA RTX 4090 GPU, running Ubuntu 22.04, and using PyTorch 2.0 as the deep learning framework. The test dataset used a publicly available encrypted malicious traffic dataset, containing encrypted traffic samples and normal HTTPS traffic samples from multiple malware families. The dataset contained a total of 500,000 encrypted session records, with malicious samples accounting for approximately 15%. To ensure the fairness and reproducibility of the experiment, the dataset was divided into training, validation, and test sets in an 8:1:1 ratio, and all experimental results were obtained on the test set.

[0109] Compared with the GRU-DNN model used in Chinese invention CN115589314A, the technical solution of this invention achieves significant improvements in multiple evaluation metrics. In terms of detection accuracy, the solution of this invention reaches 97.8%, which is comparable to the 98% accuracy of CN115589314A. In terms of recall, the solution of this invention reaches 96.5%, slightly lower than the 99.8% recall of CN115589314A, but still at a high level. In the generalization ability test against new attack variants, the accuracy of the solution of this invention is 94.2%, an improvement of approximately 8 percentage points compared to the accuracy of CN115589314A in cross-time period tests, indicating that the multi-dimensional feature extraction and self-attention encoding mechanism of this invention effectively enhance the model's generalization ability. Furthermore, the solution of this invention achieves an F1 score of 97.1% and a precision of 97.8%, these metrics comprehensively reflect the excellent performance of the model in the malicious traffic identification task.

[0110] Regarding certificate risk identification, the certificate reputation assessment module 4 of this invention can accurately identify self-signed certificates, short-term certificates, and certificates from non-authoritative institutions, achieving an accuracy rate of 99.2%, filling the gap in CN115589314A's lack of certificate risk assessment capabilities. In terms of processing efficiency, the average processing time for a single encrypted session is 12 milliseconds, meeting the performance requirements for real-time detection. Regarding resource consumption, the GPU memory usage during the model inference phase is approximately 2GB, and the CPU utilization remains below 30%, indicating that the solution has good resource efficiency and is suitable for deployment on standard network security equipment.

[0111] The embodiments of the present invention are not limited to the specific embodiments described above. Those skilled in the art can make various equivalent changes or substitutions based on the technical solutions of the present invention, and all such changes or substitutions should be included within the protection scope of the present invention.

Claims

1. A method for identifying malicious behavior in encrypted traffic, characterized in that, include: The metadata extraction module acquires encrypted network traffic data. Without decrypting the traffic content, it extracts handshake message feature data from the TLS handshake phase, certificate chain data from the certificate exchange phase, and traffic statistics sequence from the session transmission phase. The handshake message feature data includes a ClientHello extended field sequence, a cipher suite list, and a set of supported version numbers. The certificate chain data includes certificate validity information, issuing authority identifier, and a list of alternative subject names. The traffic statistics sequence includes a packet size sequence, packet interval sequence, and uplink / downlink byte statistics. The feature encoding module receives the handshake message feature data, the certificate chain data, and the traffic statistics sequence. It assigns positional information to each element in the variable-length sequence using positional encoding and calculates the sequence using a multi-head attention mechanism. The association weights between elements are used to weight and aggregate the sequence elements to generate TLS handshake feature vectors, certificate attribute feature vectors, and traffic statistics feature vectors. The deep classification module receives the TLS handshake feature vectors, certificate attribute feature vectors, and traffic statistics feature vectors, constructs a fusion input vector by concatenating the features, and inputs the fusion input vector into a classification network containing multiple self-attention encoding layers to output a behavior classification probability vector for a preset malicious behavior category. The certificate reputation assessment module receives the certificate chain data, calculates the certificate anomaly metric based on three dimensions: certificate validity period, issuing authority credibility, and number of alternative subject names, generates a certificate reputation score based on the certificate anomaly metric, and generates risk label information for certificates that meet preset risk conditions. The collaborative decision-making module receives the behavior classification probability vector, the certificate reputation score, and the risk labeling information. Based on the comparison result between the maximum probability value in the behavior classification probability vector and the preset confidence threshold, it determines the behavior type. It then performs a weighted fusion calculation of the certificate reputation score and the behavior type determination to calculate the session risk score, and outputs the session risk score, the abnormal behavior type determination result, and the list of suspicious certificate fingerprints.

2. The method for identifying malicious behavior in encrypted traffic according to claim 1, characterized in that, The extraction of handshake message feature data includes: extracting extended field type identifier sequences and corresponding extended field length sequences from the ClientHello message, extracting suite identifier sequences from the cipher suite list, and extracting protocol version identifier sequences from the supported version number set.

3. The method for identifying malicious behavior in encrypted traffic according to claim 1, characterized in that, The extraction of the traffic statistics sequence includes: statistically analyzing the distribution of data packet size in units of a preset time window, calculating the arrival time interval of adjacent data packets to form a packet interval time sequence, calculating the ratio of uplink bytes to downlink bytes, and identifying burst transmission pattern sequences formed by continuous high-volume transmission.

4. The method for identifying malicious behavior in encrypted traffic according to claim 1, characterized in that, The certificate reputation assessment module generates risk label information for certificates that meet at least one of the following preset risk conditions: the certificate is a self-signed certificate, the certificate validity period is less than a preset validity period threshold, and the certificate issuing authority is not in the preset trusted issuing authority list.

5. The method for identifying malicious behavior in encrypted traffic according to claim 1, characterized in that, The feature encoding module uses position encoding to assign position information to each element in a variable-length sequence, which includes: generating a position encoding vector for the i-th element in the sequence, wherein the even-numbered dimensions of the position encoding vector are calculated using a sine function and the odd-numbered dimensions are calculated using a cosine function, and the position encoding vector is added to the original representation vector of the corresponding element to obtain the position-enhanced representation.

6. The method for identifying malicious behavior in encrypted traffic according to claim 1, characterized in that, The multi-head attention mechanism calculates the association weights between sequence elements by: transforming the input sequence through query transformation, key transformation, and value transformation to obtain query vector group, key vector group, and value vector group respectively; calculating the dot product of the query vector and the key vector and scaling it to obtain attention weights; and multiplying the attention weights with the value vectors and concatenating the outputs of multiple attention heads to obtain multi-head attention output.

7. The method for identifying malicious behavior in encrypted traffic according to claim 1, characterized in that, Each self-attention encoding layer of the classification network in the deep classification module includes a multi-head self-attention sub-layer, a first residual connection sub-layer, a feedforward neural network sub-layer, and a second residual connection sub-layer connected in sequence. The feedforward neural network sub-layer contains two linear transformation layers and an activation function layer located between the two linear transformation layers.

8. The method for identifying malicious behavior in encrypted traffic according to claim 1, characterized in that, The preset malicious behavior categories include C2 communication, abuse of encrypted tunnels, data leakage, and malware update channels.

9. The method for identifying malicious behavior in encrypted traffic according to claim 1, characterized in that, The collaborative decision-making module calculates the session risk score by weighting and fusing the certificate reputation score and the behavior type determination. This includes: when the behavior type is determined to be malicious, multiplying the behavior classification probability by a preset behavior weight to obtain a behavior risk component, multiplying the complement of the certificate reputation score by a preset certificate weight to obtain a certificate risk component, and adding the behavior risk component and the certificate risk component to obtain the session risk score; when the behavior type is determined to be uncertain, multiplying the maximum probability value by a preset uncertainty weight to obtain an uncertain risk component, multiplying the complement of the certificate reputation score by a preset certificate weight to obtain a certificate risk component, and adding the uncertain risk component and the certificate risk component to obtain the session risk score.

10. A system for identifying malicious behavior in encrypted traffic, used to implement the method for identifying malicious behavior in encrypted traffic as described in any one of claims 1-9, characterized in that, include: The metadata extraction module is used to obtain encrypted network traffic data. Without decrypting the traffic content, it extracts handshake message feature data from the TLS handshake phase, certificate chain data from the certificate exchange phase, and traffic statistical sequences from the session transmission phase. The feature encoding module receives the handshake message feature data, the certificate chain data, and the traffic statistics sequence. It uses positional encoding and a multi-head attention mechanism to convert the variable-length sequence into a fixed-dimensional vector, generating a TLS handshake feature vector, a certificate attribute feature vector, and a traffic statistics feature vector. The deep classification module receives the TLS handshake feature vector, the certificate attribute feature vector, and the traffic statistics feature vector. It uses a classification network containing multiple self-attention encoding layers to output a behavior classification probability vector for a preset malicious behavior category. The certificate reputation evaluation module receives the certificate chain data, calculates a certificate reputation score based on the certificate validity period, issuing authority credibility, and the number of alternative subject names, and generates risk label information for certificates that meet preset risk conditions. The collaborative decision-making module is used to receive the behavior classification probability vector, the certificate reputation score and the risk labeling information, calculate the session risk score based on weighted fusion, and output the session risk score, the abnormal behavior type determination result and the list of suspicious certificate fingerprints.

Images