Network malicious traffic detection and analysis method and device, electronic equipment and readable storage medium

Abstract

The invention provides a malicious network traffic detection and analysis method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring network traffic data; performing multi-dimensional feature extraction on the network traffic data, and dividing the extracted multi-dimensional features into a plurality of semantic domains; analyzing the state distribution of each semantic domain in a preset time window, and obtaining a Hessian matrix of the global entropy of the semantic domain according to the dynamic entropy of each semantic domain; according to the Hessian matrix, screening first malicious traffic data in the network traffic data; clustering other network traffic data except the malicious traffic data in the network traffic data by using a DBSCAN algorithm to obtain at least one traffic cluster and abnormal noise points; screening second malicious traffic data in the network traffic data from the traffic clusters and the abnormal noise points according to the feature portraits of the traffic clusters and the abnormal noise points; and adjusting a traffic management strategy and / or a screening rule of the first malicious traffic data according to the traffic cluster and the feature portrait of the abnormal noise point.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present disclosure relate to the technical field of network security, and in particular, to a network malicious traffic detection analysis method and device, an electronic device, and a readable storage medium. BACKGROUND

[0002] It should be noted that the above introduction to the technical background is only for the convenience of clearly and completely describing the technical solutions of the present disclosure, and for the convenience of understanding by those skilled in the art. The above technical solutions cannot be considered as known to those skilled in the art merely because they are described in the background section of the present disclosure.

[0003] With the deep popularization of encryption protocols such as HTTPS, TLS / SSL, traditional content detection security devices (such as signature-dependent IDS / IPS, DPI firewall) are facing the challenge of a cliff-like drop in traffic visibility.

[0004] In the related art, network malicious traffic detection methods mainly rely on the behavior characteristics or metadata of encrypted traffic. However, the analysis of data in the related art is often rough, and it is difficult to capture complex and subtle behavior anomalies in encrypted traffic. This makes carefully designed covert attacks, such as through malware communication, command and control channels, or data theft, etc., can easily hide in encrypted tunnels, significantly reducing the effectiveness of existing security monitoring systems. That is, the related art lacks an effective identification mechanism based on the behavior characteristics or fine-grained metadata of encrypted traffic. SUMMARY

[0005] In view of this, the purpose of one or more embodiments of the present disclosure is to propose a network malicious traffic detection analysis method, device, electronic device, and readable storage medium to solve the problems in the background art.

[0006] Based on the above purpose, one or more embodiments of the present disclosure provide a network malicious traffic detection analysis method. It includes:

[0007] Obtaining network traffic data, the network traffic data including encrypted network traffic based on a TCP protocol;

[0008] Performing multi-dimensional feature extraction on the network traffic data, and dividing the extracted multi-dimensional features into a plurality of semantic domains;

[0009] Analyzing the state distribution of each semantic domain within a preset time window, and obtaining a Hessian matrix of the global entropy of the semantic domain according to the dynamic entropy of each semantic domain;

[0010] screening first malicious traffic data in the network traffic data according to the Hessian matrix, the first malicious traffic representing malicious traffic data with behavior pattern mutation;

[0011] applying a DBSCAN algorithm to cluster other network traffic data in the network traffic data except the malicious traffic data to obtain at least one traffic cluster and an abnormal noise point;

[0012] screening second malicious traffic data in the network traffic data from the traffic cluster and the abnormal noise point according to feature portraits of the traffic cluster and the abnormal noise point, the second malicious traffic data representing potential malicious traffic with statistical outliers;

[0013] adjusting a traffic management strategy and / or a screening rule of the first malicious traffic data according to the feature portraits of the traffic cluster and the abnormal noise point.

[0014] Optionally, the multi-dimensional features of the network traffic data are divided into multiple domains of network protocols, time sequences, traffic behaviors, traffic features and encryption processes according to semantics.

[0015] Optionally, analyzing a state distribution of each of the semantic domains within a preset time window to obtain a Hessian matrix of global entropy of each of the semantic domains, including:

[0016] discretizing the multi-dimensional features of the network traffic data into intra-domain states;

[0017] judging whether the network traffic data activates a malicious behavior pattern in a malicious behavior pattern domain according to the intra-domain states of the semantic domains, the malicious behavior pattern including at least one of a suspicious beacon pattern, a scanning and investigation pattern, an abnormal data exfiltration pattern, an abnormal payload download pattern, an encryption negotiation pattern and an unexpected behavior deviation pattern;

[0018] generating an intra-domain state of the malicious behavior pattern domain according to an activation state of the malicious behavior pattern;

[0019] obtaining a state distribution Shannon entropy of each of the semantic domains and the malicious behavior pattern domain according to the intra-domain states of each of the semantic domains and the malicious behavior pattern domain within a preset time window;

[0020] weighting and aggregating the state distribution Shannon entropy according to importance of each of the semantic domains and the malicious behavior pattern domain to obtain a global entropy sequence, elements in the global entropy sequence being arranged in time sequence;

[0021] obtaining the Hessian matrix according to the global entropy sequence.

[0022] Optionally, the screening the first malicious traffic data in the network traffic data according to the Hessian matrix comprises:

[0023] According to the Hessian matrix, a Frobenius norm of the Hessian matrix is obtained, and the Hessian matrix and the Frobenius norm represent a change rate and a complexity of a state distribution Shannon entropy of each semantic domain;

[0024] According to the Hessian matrix and the Frobenius norm, an anomaly score is obtained, and the anomaly score represents a change degree of the state distribution Shannon entropy of each semantic domain in a current time window;

[0025] In response to determining that the anomaly score is greater than or equal to a preset threshold, it is determined that malicious traffic exists in the network traffic data;

[0026] The network traffic data is analyzed to determine the first malicious traffic data.

[0027] Optionally, the screening the second malicious traffic data in the network traffic data according to the feature portraits of the traffic clusters and the abnormal noise points comprises:

[0028] Feature statistics are performed on the traffic clusters and the abnormal noise points;

[0029] According to the features of the traffic clusters and the features of the abnormal noise points, a first feature portrait of the traffic clusters and a second feature portrait of the abnormal noise points are obtained, and the feature portraits are used to describe a representative traffic mode and a potential security implication;

[0030] The second malicious traffic data in the network traffic data is screened according to the first feature portrait and the second feature portrait.

[0031] Optionally, the adjusting a traffic management strategy and / or a screening rule of the first malicious traffic data according to the feature portraits of the traffic clusters and the abnormal noise points comprises:

[0032] In response to any of the first feature portraits indicating that the traffic cluster has a violation behavior, a traffic management strategy is formulated according to the first feature portrait, and the traffic management strategy comprises at least one of a network access control strategy and a bandwidth limitation strategy;

[0033] In response to a same type of risk signal appearing in multiple traffic clusters or the abnormal noise points, a screening rule of the first malicious traffic data is adjusted according to a type of the risk signal.

[0034] Optionally, feature extraction is performed on the network traffic data from multiple levels of flow, packet, TCP protocol and TLS handshake process.

[0035] Based on the same inventive concept, one or more embodiments of the present disclosure further provide a network malicious traffic detection analysis device, comprising:

[0036] a data acquisition module configured to acquire network traffic data, the network traffic data comprising encrypted network traffic based on TCP protocol;

[0037] a feature extraction module configured to perform multi-dimensional feature extraction on the network traffic data, and divide the extracted multi-dimensional features into multiple semantic domains;

[0038] a first calculation module configured to analyze the state distribution of each semantic domain within a preset time window, and obtain a Hessian matrix of global entropy of the semantic domain according to the dynamic entropy of each semantic domain;

[0039] a first screening module configured to screen first malicious traffic data in the network traffic data according to the Hessian matrix, the first malicious traffic data representing malicious traffic data with abrupt behavior pattern mutation;

[0040] a second calculation module configured to apply a DBSCAN algorithm to cluster other network traffic data in the network traffic data except the malicious traffic data, to obtain at least one traffic cluster and an abnormal noise point;

[0041] a second screening module configured to screen second malicious traffic data in the network traffic data from the traffic cluster and the abnormal noise point according to the feature portrait of the traffic cluster and the abnormal noise point, the second malicious traffic data representing potential malicious traffic with statistical outliers;

[0042] adjust a traffic management strategy and / or a screening rule of the first malicious traffic data according to the feature portrait of the traffic cluster and the abnormal noise point.

[0043] Optionally, the feature extraction module is specifically configured to:

[0044] divide the multi-dimensional features of the network traffic data into multiple domains of network protocol, time sequence, traffic behavior, traffic feature and encryption process according to semantics.

[0045] Optionally, the first calculation module is specifically configured to:

[0046] discretize the multi-dimensional features of the network traffic data into in-domain states;

[0047] determine whether the network traffic data activates a malicious behavior pattern in a malicious behavior pattern domain according to a domain state of the semantic domain, the malicious behavior pattern including at least one of a suspicious beacon pattern, a scanning probe pattern, an abnormal data exfiltration pattern, an abnormal payload download pattern, an encrypted negotiation pattern, and an unexpected behavior deviation pattern;

[0048] generate a domain state of the malicious behavior pattern domain according to an activation state of the malicious behavior pattern;

[0049] obtain a state distribution Shannon entropy of each of the semantic domain and the malicious behavior pattern domain according to a domain state of each of the semantic domain and the malicious behavior pattern domain within a preset time window;

[0050] weight and aggregate the state distribution Shannon entropy according to importance of each of the semantic domain and the malicious behavior pattern domain to obtain a global entropy sequence, elements in the global entropy sequence being arranged in time sequence;

[0051] obtain the Hessian matrix according to the global entropy sequence.

[0052] Optionally, the first screening module is specifically configured to:

[0053] obtain a Frobenius norm of the Hessian matrix according to the Hessian matrix, the Hessian matrix and the Frobenius norm representing a change rate and a complexity of the state distribution Shannon entropy of each semantic domain;

[0054] obtain an anomaly score according to the Hessian matrix and the Frobenius norm, the anomaly score representing a change degree of the state distribution Shannon entropy of each semantic domain within a current time window;

[0055] determine that there is malicious traffic in the network traffic data in response to determining that the anomaly score is greater than or equal to a preset threshold;

[0056] analyze the network traffic data to determine first malicious traffic data.

[0057] Optionally, the second screening module is specifically configured to:

[0058] perform feature statistics on the traffic cluster and the abnormal noise point;

[0059] obtain a first feature portrait of the traffic cluster and a second feature portrait of the abnormal noise point according to features of the traffic cluster and features of the abnormal noise point, the feature portrait being used to describe a represented traffic pattern and a potential security implication;

[0060] Screening second malicious traffic data in the network traffic data according to the first feature portrait and the second feature portrait.

[0061] Optionally, the policy optimization module 17 is specifically configured to:

[0062] In response to any of the first feature portraits indicating that the traffic cluster exists a violation behavior, formulating a traffic management policy according to the first feature portrait, the traffic management policy including at least one of a network access control policy and a bandwidth limitation policy;

[0063] In response to the same type of risk signal appearing in multiple traffic clusters or the abnormal noise points, adjusting the screening rule of the first malicious traffic data according to the type of the risk signal.

[0064] Optionally, the feature extraction module is specifically configured to:

[0065] Extracting features from multiple levels of the network traffic data, including flow, packet, TCP protocol and TLS handshake process.

[0066] Based on the same inventive concept, one or more embodiments of the present disclosure further provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the network malicious traffic detection and analysis method according to any one of the above.

[0067] Based on the same inventive concept, one or more embodiments of the present disclosure further provide a non-transitory computer readable storage medium, which stores computer instructions for causing the computer to execute the network malicious traffic detection and analysis method according to any one of the above.

[0068] As can be seen from the above, the network malicious traffic detection and analysis method provided by one or more embodiments of the present disclosure preliminarily screens malicious traffic by mapping features extracted from multiple levels to multiple semantic domains and analyzing dynamic entropy of discrete states of each semantic domain, and further screens malicious traffic based on a DBSCAN clustering algorithm. The present disclosure combines the real-time anomaly detection capability of dynamic entropy with the deep pattern mining and outlier discovery capability of clustering, and can effectively analyze encrypted traffic without decryption; through continuous analysis and utilization of the identified malicious features, benign pattern portraits and abnormal noise points, a learning and improvement closed loop is formed, the parameters and rules of the detection model are continuously optimized, the accuracy of malicious activity identification is significantly improved, the depth of understanding of network behavior patterns is improved, and data support is provided for security policy optimization, thereby realizing efficient and accurate monitoring of modern encrypted network environment.

[0069] The network malicious traffic detection analysis device, the electronic device, and the computer readable storage medium provided by the present disclosure can implement the steps of the network malicious traffic detection analysis method, and therefore have the beneficial effects of the network malicious traffic detection analysis method. BRIEF DESCRIPTION OF DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the one or more embodiments of the present disclosure or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only one or more embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0071] Figure 1 Flowchart of the network malicious traffic detection analysis method of the one or more embodiments of the present disclosure;

[0072] Figure 2 Structure diagram of the network malicious traffic detection analysis device of the one or more embodiments of the present disclosure;

[0073] Figure 3 Structure diagram of multi-domain modeling and state discretization of the one or more embodiments of the present disclosure;

[0074] Figure 4 Flowchart of the malicious encrypted traffic identification process based on multi-domain dynamic entropy analysis of the one or more embodiments of the present disclosure;

[0075] Figure 5 Flowchart of the DBSCAN clustering and noise point marking algorithm of the one or more embodiments of the present disclosure;

[0076] Figure 6 Flowchart of the encrypted traffic analysis method of the one or more embodiments of the present disclosure;

[0077] Figure 7 Hardware structure diagram of the electronic device of the one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0078] In order to make the purposes, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to specific embodiments and drawings.

[0079] It should be noted that the technical terms or scientific terms used in one or more embodiments of the present disclosure should be understood as the general meaning understood by those skilled in the art to which the present disclosure belongs, unless otherwise defined. The terms "first", "second", and the like used in one or more embodiments of the present disclosure do not represent any order, quantity or importance, but are only used to distinguish different components. The terms "include" or "contain" and the like mean that the elements or objects before the terms cover the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connected" or "connected" and the like are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to represent relative positional relationships, and when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0080] As described in the background section, with the widespread use of encryption protocols, most of the content of network traffic becomes invisible to traditional content detection-based security devices.

[0081] In the related art, the security performance of network traffic is detected by decrypting network traffic, or directly analyzing the features or metadata of encrypted traffic.

[0082] However, in practice, network traffic decryption often faces huge performance overhead, serious privacy compliance risks, and multiple challenges such as certificate management difficulties and system compatibility problems.

[0083] And the feature extraction or metadata analysis process of encrypted traffic in the related art is often rough, and it is difficult to capture complex and subtle behavior anomalies in encrypted traffic. This makes carefully designed covert attacks, such as through malware communication, command and control channels, or data theft, can easily hide in encrypted tunnels, significantly reducing the effectiveness of existing security monitoring systems.

[0084] That is, the related art is difficult to effectively distinguish between malicious activities and various normal business communications in encrypted traffic without decryption, because they lack effective identification mechanisms based on the behavior features or refined metadata of encrypted traffic.

[0085] Thus, the present disclosure provides a network malicious traffic detection analysis method, which first collects traffic data and extracts multi-dimensional features to divide into semantic domains, generates a Hessian matrix of global entropy by analyzing the state distribution of each semantic domain in a preset time window, screens out the first type of malicious traffic with behavior pattern mutation, and then clusters the remaining traffic using the DBSCAN algorithm to obtain traffic clusters and abnormal noise points, and screens out the second type of potential malicious traffic based on the statistical outliers of the feature portraits. Compared with the prior art, the scheme realizes fine modeling through multi-dimensional feature extraction and semantic domain division, captures behavior mutation using the Hessian matrix, and combines DBSCAN clustering to mine statistical outliers, forming a two-stage detection framework without decrypting the traffic throughout. Effectively solve the problems of traditional methods such as extensive feature extraction, inability to capture subtle anomalies, dependence on decryption, and single detection dimension, and improve the recognition ability of hidden attacks in encrypted traffic.

[0086] Reference Figure 1 The network malicious traffic detection analysis method of one or more embodiments of the present disclosure includes the following steps:

[0087] Step S101: Obtain network traffic data, wherein the network traffic data includes encrypted network traffic based on the TCP protocol;

[0088] Step S102: Perform multi-dimensional feature extraction on the network traffic data, and divide the extracted multi-dimensional features into multiple semantic domains;

[0089] Step S103: Analyze the state distribution of each semantic domain in a preset time window to obtain the Hessian matrix of the global entropy of the semantic domain;

[0090] Step S104: According to the Hessian matrix, screen the first malicious traffic data in the network traffic data, wherein the first malicious traffic represents malicious traffic data with behavior pattern mutation;

[0091] Step S105: Apply the DBSCAN algorithm to cluster other network traffic data in the network traffic data except the malicious traffic data to obtain at least one traffic cluster and abnormal noise points;

[0092] Step S106: According to the feature portraits of the traffic cluster and the abnormal noise points, screen the second malicious traffic data in the network traffic data from the traffic cluster and the abnormal noise points, wherein the second malicious traffic data represents potential malicious traffic with statistical outliers;

[0093] Step S107, according to the feature portraits of the traffic cluster and the abnormal noise points, adjust the traffic management strategy and / or the screening rules of the first malicious traffic data.

[0094] The network traffic data in step S101 includes encrypted network traffic based on the TCP protocol. This encrypted network traffic based on the TCP protocol can be obtained by filtering from the original network traffic using a filtering method. In some examples of this disclosure, filtering encrypted network traffic based on the TCP protocol from the original network traffic may include:

[0095] Receive raw network traffic. Capture raw data packets on the network using a network interface card (NIC) or traffic mirroring / splitting device. This raw network traffic can be defined as T. raw ={pkt1,pkt2,…,pkt n}, where pkt i This represents the i-th data packet, and n represents the total number of data packets.

[0096] Filter encrypted network traffic based on the TCP protocol and derive its structured representation. For T raw Each data packet pkt i Define the filtering function f(pkt) i ).

[0097]

[0098] Wherein, P(pkt) i ) indicates the data packet pkt i Associated port number information, P known S(pkt) is a predefined set of known port numbers, containing port numbers associated with a specific application protocol. i ) represents the TCP payload length pkt i The size, threshold is a predefined threshold; D(pkt) i The ) indicates the direction of data packet flow, and down indicates the inbound direction (entering the monitored network or system). Through filtering, the data packet set T′={pkt} is obtained. i ∈T raw |f(pkt i = Accept}.

[0099] The above data packet set T′ can be derived into a structured representation as flow. k .

[0100] flow k ={pkt i ∈T′|5-tuple(pkt i ) = 5-tuple k}

[0101] 5-tuple(pkt i) = (Source IP Address, Destination IP Address, Source Port, Destination Port, Protocol Number)

[0102] That is, the five-tuple (5-tuple) is used as a unique identifier of a TCP flow, and the protocol number Protocol Number = 6 corresponds to the TCP protocol. k represents the kth five-tuple combination appearing in T ′ , k is used to distinguish different flows identified in T ′ , and is further used to extract flow-level features describing the entire connection behavior in the subsequent steps.

[0103] In this way, the encrypted network traffic based on the TCP protocol can be organized into logical TCP flows according to the five-tuple (source / destination IP, source / destination port, protocol number).

[0104] In some implementations of the present disclosure, step S102 can extract features from multiple levels in the flow, packet, TCP protocol, and TLS handshake process.

[0105] Specifically, in some implementations of the present disclosure, the flow-level features can include: the difference between the timestamp of the last packet and the timestamp of the first packet in the flow, the total number of packets constituting the flow, the total number of bytes of the flow, the average packet size, the packet size standard deviation, the average packet arrival interval, and the packet arrival interval standard deviation.

[0106] In some examples of the present disclosure, the difference between the timestamp of the last packet and the timestamp of the first packet in the flow, i.e., the total duration of a TCP connection from establishment to end (or to the observation end), can be calculated by the following formula:

[0107] session_duration(flow k ) = timestamp(pkt last ) - timestamp(pkt first )

[0108] Wherein, timestamp represents the timestamp, pkt last represents the last packet, pkt first represents the first packet,

[0109] The total number of packets constituting the flow can be calculated by the following formula: total_packets(flow k ) = |flowk This feature can reflect the frequency of communication interactions or the degree of data segmentation.

[0110] total_bytes(flow k ) can be calculated based on uplink and downlink traffic respectively. The total byte count calculation formula of uplink traffic can be:

[0111]

[0112] The total byte count calculation formula of downlink traffic can be:

[0113]

[0114] The calculation formula of average packet size can be:

[0115] avg_pkt_size(flow k ) = total_bytes(flow k ) / total_packets(flow k )

[0116] In some examples of the present disclosure, the average packet size of uplink traffic and the average packet size of downlink traffic can also be calculated respectively according to the total byte count of uplink traffic and the total byte count of downlink traffic.

[0117] The calculation formula of packet size standard deviation can be:

[0118]

[0119] In some examples of the present disclosure, the packet size standard deviation of uplink traffic and the packet size standard deviation of downlink traffic can also be calculated respectively according to the total byte count of uplink traffic and the total byte count of downlink traffic. The packet size standard deviation feature can be used to measure the degree of change or dispersion of TCP load size in a particular direction.

[0120] The calculation formula of average packet inter-arrival time can be:

[0121]

[0122] The calculation formula of packet inter-arrival time standard deviation can be:

[0123]

[0124] where IAT i = timestamp(pkt i+1 ) - timestamp(pkt i ),

[0125] In some implementations of the disclosure, the packet-level features can include: a timestamp timestamp(pkt i ), a TCP payload size S(pkt i ), and a traffic direction Flow_Direction D(pkt i ). These features can be directly extracted from a single packet pkt i .

[0126] In some implementations of the disclosure, the TCP protocol level features can include: a sequence number seq(pkt i ), an acknowledgement number ack(pkt i ), a round-trip time RTT(pkt i ) calculated using TCP timestamps or data-acknowledgement pairing, TCP flags TCPflags(pkt i ), and a window size WindowSize(pkt i ). The values of the TCP flags can include SYN, ACK, FIN, RST, PSH, URG, etc. The window size represents the number of bytes that the buffer of the receiving side can receive, which can be directly obtained from the TCP header.

[0127] In some examples of the disclosure, the formula for calculating the round-trip time using TCP timestamps or data-acknowledgement pairing can be:

[0128] RTT(pkt i ) = timestamp(ack(pkt i )) - timestamp(pkt i )

[0129] wherein timestamp(pkt i ) is the timestamp of the packet pkt i , and timestamp(ack(pkt i )) is the timestamp of the corresponding acknowledgement packet of the packet pkt i .

[0130] In some implementations of the disclosure, the features extracted in the TLS handshake process can include: a negotiated TLS version TLsversion(flow k ), a server-selected cipher suite / JA3 fingerprint (client list) / JA3S fingerprint (server selection): Cipher_Suites(flow k ), a handshake message length Client_Hello_Length(flow k ), and Server_Hello_Length(flowk ), the original binary data of the complete certificate chain Certificate_Chain_Full(flow k ), based on the first certificate (terminal entity certificate) in Certificate_Chain_Full(flow k ) to obtain the core certificate Parsed_Certificate_Features(primary_cert) in the certificate chain and the server name indication SNI(flow k ).

[0131] Among them, the client supported TLS version can be extracted from the Client Hello message when parsing the handshake message, or the server selected TLS version can be extracted from the Server Hello message.

[0132] In some implementations of the present disclosure, after feature extraction, feature standardization and encoding can also be performed. The purpose of doing so is to eliminate the scale difference between features and convert the classification features, in order to prepare for model and entropy calculation.

[0133] In some examples of the present disclosure, Z-score standardization can be applied to numerical features. Specifically, the mean μ and standard deviation σ of the extracted numerical features X can be calculated, and then standardized: X norm =(X-μ) / σ, so as to conform to the standard normal distribution (mean 0, standard deviation 1). For classification features (such as TLSversion(flow k ), Cipher_Suites(flow k ) and the like), they need to be converted into numerical form suitable for machine learning models. One-Hot Encoding or Target Encoding can be selected according to actual conditions.

[0134] In some implementations of the present disclosure, the final output is the feature dataset ProcessedFeatureSet after standardization and encoding.

[0135] In some implementations of the present disclosure, the semantic domain can include multiple network protocols, timing, traffic behavior, traffic features and encryption processes. The extracted multi-dimensional features are divided into multiple semantic domains, in order to prepare for the first stage of malicious traffic detection analysis (cross-domain entropy feature detection and filtering of potential malicious encryption flow).

[0136] The present disclosure also introduces a malicious behavior pattern domain to realize the key mapping of feature discretization to the semantic domain and specific malicious behavior pattern determination.

[0137] In some implementations of the present disclosure, the above-mentioned malicious behavior pattern domain includes at least one of a suspicious beacon pattern, a scanning probe pattern, an abnormal data exfiltration pattern, an abnormal payload download pattern, an encrypted negotiation pattern, and an unexpected behavior deviation pattern.

[0138] In some implementations of the present disclosure, the activation logic of each pattern can determine whether to trigger based on a comparison of a specific combination or weighted rarity score of discrete states of the above-mentioned network protocol, timing, traffic behavior, traffic feature, encryption process semantic domains within a specified time window with a corresponding threshold value.

[0139] In some examples of the present disclosure, the network protocol domain can discretize the protocol, port number information, negotiated TLS version, and packet direction and size of a specific packet. That is, the network protocol domain can be represented as d1={P(pkt i ),TLSversion(flow k ),S(pkt i ),D(pkt i )},d1 represents the network protocol domain.

[0140] In implementations of the present disclosure, the timing domain can focus on time-related features, reflecting the timing behavior characteristics of traffic. That is, the timing domain can be represented as d2={session_duration(flow k ),avg_interarrival_time(flow k ),d2 represents the timing domain.

[0141] In some examples of the present disclosure, the traffic behavior domain can describe the behavior pattern of traffic, and can focus on the size, retransmission, window size, and other characteristics of traffic. That is, the traffic behavior domain can be represented as d3={avg_pkt_size(flow k ),pkt_size_stddev(flow k ),d3 represents the traffic behavior domain.

[0142] In some examples of the present disclosure, the traffic feature domain can discretize the total number of bytes, the total number of packets, and the uplink / downlink byte ratio state of the flow. That is, the traffic feature domain can be represented as d4={total_packets(flow k ),total_bytes(flow k ),state(flow k )},wherein d4 represents the traffic feature domain, and state(flow k ) represents the uplink / downlink byte ratio state. The calculation steps are as follows: wherein The proportion value proportion_up(flow k ) will be between 0 and 1, reflecting the main direction of the flow, and then state(flow k ) is discretized into different state spaces according to a predefined threshold.

[0143] In some examples of the present disclosure, the encryption process domain can discretize features related to the encryption process and the TLS handshake (such as TLS version, cipher suite, certificate length, SNI, JA3 fingerprint, etc.). That is, the encryption process domain can be represented as d5 = {Cipher_Suites(flow k ), Hello_Length_State(flow k )}. Wherein d5 represents the encryption process domain.

[0144] All the above features need to be discretized into different state spaces when discretized.

[0145] As shown in Figure 3 , in one embodiment of the present disclosure, the semantic domain includes network protocol, timing, traffic behavior, traffic features, and encryption process, and is defined as d1, d2, d3, d4, d5, and d6, respectively. In this embodiment, the port information, TLS version, TCP payload size, and traffic direction features are discretized into the network protocol domain; the session length, average packet arrival time, and packet arrival interval standard deviation features are discretized into the timing domain; the average packet size, packet size standard deviation, round-trip time, TCP flag bit, and window size features are discretized into the traffic behavior domain; the total packet number, total byte number, and uplink / downlink byte proportion features are discretized into the traffic feature domain; and the cipher suite, handshake message length, certificate chain state, and SNI features are discretized into the encryption process domain.

[0146] Then, according to the discretized state set of the network protocol, timing, traffic behavior, traffic feature, and encryption process domains within a preset window, this embodiment determines whether to activate the malicious behavior mode in the malicious behavior mode domain.

[0147] In this embodiment, the malicious behavior mode includes suspicious beacon mode, scanning and investigation mode, abnormal data export mode, abnormal large capacity download mode, can encrypt negotiation mode, and unexpected behavior deviation mode.

[0148] In some examples of the present disclosure, the activation condition of the suspicious beacon mode is:

[0149]

[0150] The triggering logic rule A can be defined as: in the active flows in the time window Δt, the standard deviation of the data packet arrival time interval is very low, the total number of bytes transmitted by the entire flow is very small, and the total number of data packets contained in the entire flow is also very small. The proportion of the state of the flow > Th_Beacon_Ratio1, Th_Beacon_Ratio1 represents the minimum threshold of the proportion of the flow determined as “beacon-like” behavior, which can be set by empirical knowledge or baseline analysis.

[0151] The logic rule B can be defined as: in the newly created flows or active flows in the time window Δt, the proportion of the state of the flow in which the SNI is marked as known malicious or the SNI is suspected to be generated by a domain name generation algorithm (DGA) or the characteristics of the resolution server certificate are found to be marked as known malicious > Th_Beacon_Ratio2 or the absolute number of such flows > Th_Beacon_Count, Th_Beacon_Ratio2 is the minimum threshold of the proportion of the flow connected to the known malicious target, and Th_Beacon_Count is the minimum absolute number of the flow connected to the known malicious target.

[0152] In some examples of the present disclosure, the activation condition of the scanning reconnaissance pattern is:

[0153] Scanning_Reconnaissance_Pattern_Active(Δt) simultaneously satisfies the following conditions.

[0154] The conditions can include:

[0155] Newly created flow rate condition: N new_flows (t) / Δt>Th_Scan_Rate, where N new_flows (t) represents the number of newly created flows in the window t, Δt is the duration of the time window, and Th_Scan_Rate is the threshold of the newly created flow rate;

[0156] State condition: In these newly created flows, one of the following conditions is met, the proportion of the flow in which session_duration(flow k ) = State_Duration_VeryShort, total_packets(flow k ) = State_PktCnt_VeryLow, TCPflags(pkt i ) = State_Flags_HighSYN, TCPflags(pkt i ) = State_Flags_HighRST > Th_Scan_Ratio, where P state_condition (t)>Th_Scan_Ratio, where Pstate_condition (t) is the proportion of flows in the new flow that meet the above 4 state conditions, Th_Scan_Ratio is the proportion threshold.

[0157] In some examples of the present disclosure, the activation condition of the abnormal data exfiltration pattern can be:

[0158] Anomalous_Data_Exfiltration_Pattern_Active(t) is triggered if any of the following rules is met.

[0159] Rule A can be defined as: large volume or high rate upload, evaluate the overall upload behavior in the evaluation window t, meet at least one of the following, flow state state(flow k ) = Uplink Dominant and total_bytes(flow k ) = State_ByteCnt_High, the total upload byte number of the flow in the state > Th_Exfil_HighVolFlowCount, Th_Exfil_HighVolFlowCount represents the minimum threshold of the number of flows that need to be reached at the same time to trigger the "large volume concurrent upload" alarm;

[0160] Rule B can be defined as: significant upload to suspicious targets, the number of flows in the active flow in the window t that are in the following states at the same time > Th_Exfil_SuspDestFlowCount, state(flow k ) = Uplink Dominant, SNI(flow k ) = State_SNI_KnownMalicious or SNI(flow k ) = State_SNI_SuspectedDGA or SNI(flow k ) = State_SNI_PresentRare, Parsed_Certificate_Features(primary_cert) = Steate_Cert_KnownMalicious or Parsed_Certificate_Features(primary_cert) = State_Cert_SelfSigned or Parsed_Certificate_Features(primary_cert) = State_Cert_NonPublicCA, total_bytes(flow k ) = State_ByteCnt_Medium or total_bytes(flowk Suspicious_Encryption_Negotiation_Pattern_Active(t) = # of flows in active flows at window t that are in any of the following states > Th_Download_HighVolFlowCount, i.e. the minimum threshold of flows that are downloading large volumes of data simultaneously to trigger the "High Volume Download" alert.

[0161] In some examples of the present disclosure, the activation condition for the suspicious encryption negotiation pattern can be:

[0162] Suspicious_Encryption_Negotiation_Pattern_Active(t) = # of flows in active flows at window t that are in any of the following states > Th_Download_HighVolFlowCount, i.e. the minimum threshold of flows that are downloading large volumes of data simultaneously to trigger the "High Volume Download" alert.

[0163] state(flow k ) = Downlink Dominant and total_bytes(flow k ) = State_ByteCnt_High, Th_Download_HighVolFlowCount represents the minimum threshold of flows that are downloading large volumes of data simultaneously to trigger the "High Volume Download" alert.

[0164] In some examples of the present disclosure, the activation condition for the suspicious encryption negotiation pattern can be:

[0165] Suspicious_Encryption_Negotiation_Pattern_Active(t) = # of flows in new TLS flows at window t that are in any of the following states > Th_Susp_TLS_Ratio or the absolute number > Th_Susp_TLS_Count:

[0166] TLSversion(flow k ) = State_TLS_Outdated or TLSversion(flow k ) = State_SSL_Insecure, Cipher_Suites(flow k ) = State_CS_Weak or Cipher_Suites(flow k) = State_SNI_KnownMalicious or SNI(flow k ) = State_SNI_KnownMalicious or SNI(flow k ) = State_SNI_SuspectedDGA, Th_Susp_TLS_Ratio represents the minimum number of flows in the new TLS connections that exhibit suspicious negotiation features required to be reached, and Th_Susp_TLS_Count is the minimum absolute number of flows in the new TLS connections that exhibit suspicious negotiation features required to be reached.

[0167] In some examples of the present disclosure, the activation condition for the unexpected behavioral deviation pattern can be:

[0168] Unexpected_Behavioral_Deviation_Pattern_Active(t) is satisfied by computing the “rare state” score in encrypted traffic to meet a preset threshold.

[0169] Rare states and critical states can apply different weights.

[0170] For each discrete state s in d = 1 to d = 5 (e.g. State_Known_Service, State_Duration_Short, State_SC_Weak, etc., for a total of M states), compute its average probability of occurrence in all normal traffic time windows where count s is the total number of times state s occurs in all normal traffic time windows, and ∑ s′∈All States count s′is the sum of all state occurrences, so this step outputs a look-up table or dictionary P containing the normal probability baseline for M states norm (s).

[0171] Set a threshold P rare_threshold to determine which states are “rare” according to the obtained P norm (s). Those states that are highly suspicious even if they occur alone can be identified by the skilled person in the art in combination with specific circumstances, baseline analysis or empirical knowledge, which are usually directly linked to known security risks or irregular behaviors. A set of state names Critical_Rare_States is artificially defined, and different weights are assigned to critical rare states Weight_Critical and normal rare states Weight_Normal_Rare to highlight the importance of critical states. The weighted rarity score Weighted_Rarity_Score(t) is calculated as follows: s∈All States (count s (t)·Weight(s)), where if s∈Critical_Rare_States, then Weight(s) = Weight_Critical, otherwise Weight(s) = Weight_Normal_Rare. A baseline Th_Base_RarityScore is set to determine whether the weighted rarity score Weighted_Rarity_Score(t) is abnormally high. When a specific high-risk state combination is detected, the detection sensitivity is temporarily increased by adjusting the threshold, i.e. some highly suspicious state combinations are artificially defined (e.g. self-signed certificate and upload behavior, known malicious SNI and weak encryption, etc.), and if any combination rule is triggered within the window t, the dynamically adjusted threshold Th_Dynamic_RarityScore(t) = Th_Base_RarityScore × Dynamic_Factor is obtained, where Dynamic_Factor is a factor less than 1. Finally, if Weighted_Rarity_Score(t) is greater than Th_Dynamic_RarityScore(t) or Th_Base_RarityScore without dynamic adjustment, the unexpected behavior deviation mode is triggered.

[0172] It should be noted that all the above thresholds can be limited according to actual use. The applicant does not make specific limitations here.

[0173] After obtaining the discrete states of each domain (including the semantic domain and the malicious behavior pattern domain), the state distribution entropy of each domain can be calculated, and the first malicious traffic can be screened according to the state distribution entropy.

[0174] As shown in FIG. 3B, the specific process of screening the first malicious traffic can include: Figure 4

[0175] First, the state distribution entropy of each semantic domain is calculated.

[0176] In some examples of the present disclosure, the total number of occurrences of all states in domain d within a time window t is calculated as count(s, t) represents the number of occurrences of state s within time window t, and M is the total number of all possible states in the domain.

[0177] The probability of state s within time window t is calculated as

[0178] In some examples of the present disclosure, the probability calculation formula can be where p s (t) is the probability of state s within time window t.

[0179] Shannon entropy is used to measure the uncertainty of the state probability distribution within time window t. The calculation formula is where H d (t) is the Shannon entropy, and for p s (t) = 0, 0 · log2(0) is defined to avoid invalid terms in logarithmic calculation. According to the calculation formula, the state distribution Shannon entropy of each semantic domain can be obtained.

[0180] The domain entropy matrix is output, which has a size of N × T, where N represents the number of domains, and T represents the number of time windows. Each element H d (t) corresponds to the Shannon entropy of domain d within time window t.

[0181]

[0182] The domain entropy matrix of one embodiment of the present disclosure is as follows (in this embodiment, N = 6).

[0183] Then, the entropy values are processed and aggregated.

[0184] In some examples of the present disclosure, the process of processing and aggregating can include:

[0185] For each domain d, the entropy H d (t) is calculated to obtain its historical (or baseline) mean μ d and standard deviation σ d :

[0186] The entropy of each time window t is standardized using the mean and standard deviation: ​ is the entropy value of domain d in time window t after standardization.

[0187] According to the importance of each semantic domain, the weight of the semantic domain is set (dynamic weight or static weight can be set). For example, in one embodiment of the present disclosure, the weight w d is set according to prior knowledge or experience of domain experts (for example, it is considered that the encryption process domain (d = 5) and the malicious behavior pattern domain (d = 6)) to reflect the contribution of each domain to the global entropy, where In another embodiment of the present disclosure, in order to enhance the response sensitivity to recent abnormal changes, dynamic weight can be used to adjust the weight of different domains. The dynamic weight adjusts the weight value according to the entropy change in the time window, and the domain that has a larger change in the recent period will obtain a higher weight. The dynamic weight is defined in an exponentially decaying manner: where w d (t) is the dynamic weight of domain d in time window t, λ d is the decay coefficient of domain d, which controls the rate of change of the weight with time, and a larger λ d will make the weight decay faster.

[0188] On the basis of the standardized entropy value, the standardized entropy value of each domain is weighted and aggregated according to the weight of each domain. The standardized entropy value is multiplied by the weight, and the weighted sum of all domains is obtained. The global entropy H

[0189] The output of the entropy value processing is a time series of total entropy H total (t), which represents the overall information uncertainty of the system in each time window t.

[0190] Then, the entropy dynamics can be further analyzed and the anomaly score can be calculated.

[0191] Considering that the Hessian matrix can be used to measure the curvature of the entropy change, that is, the change rate in the multi-dimensional entropy space. The applicant found that by calculating the second-order partial derivative of the Hessian matrix, the complexity of the interaction between the entropy values of different domains can be obtained.

[0192] Calculate the second-order partial derivative of H total (t) with respect to each domain entropy H d (t): the elements of the Hessian matrix where H i and H j are the entropy values of two domains (such as the network protocol domain and the traffic behavior domain). Since the time window is discrete, the second-order partial derivative can be approximated by finite difference method: where ΔHi is the amount of change of domain i within the time window.

[0193] Compute the Frobenius norm of Hessian matrix (‖H(d)‖ F ). The Frobenius norm is a standard to measure the size of a matrix, which is used to calculate the complexity of Hessian matrix. This norm reflects the curvature of the entropy space, denoted as:

[0194] Compute the anomaly score AnomalyScore(t). The anomaly score quantifies the dynamic change of entropy by computing the Frobenius norm of Hessian matrix. The anomaly score is used to detect abnormal behavior: AnomalyScore(t) = ‖H(t)‖ F . A higher AnomalyScore(t) indicates that the system has a larger entropy change within this time window, which may indicate the presence of malicious activities or abnormal behaviors.

[0195] The final output of this step is the anomaly score time series.

[0196] Finally, the first malicious traffic data can be screened.

[0197] By comparing the anomaly score with the preset threshold, the first malicious traffic data can be screened, and a malicious traffic alert list can be output. If no first malicious traffic data is screened, the next stage of screening can be performed to determine the second malicious traffic data.

[0198] In some examples of the present disclosure, the threshold value θ can be determined by testing on a validation dataset containing known normal traffic and malicious traffic (e.g., ransomware traffic), and the ROC curve is usually used to find the best balance point. If AnomalyScore(t) > θ, it is considered that the traffic within the time window t is abnormal, and the traffic of the time window is marked as “suspected malicious encrypted traffic”, triggering an alarm, recording the relevant flow information for further analysis: AlertInfo(t) = {IP source , IP destination , port source , port destination ,...}, isolating or blocking the relevant connections. Finally, a malicious traffic alert or event list Malicious_Alert_List is output, which contains the relevant information of all traffic judged as malicious.

[0199] In addition to the real-time anomaly detection capability based on dynamic entropy for malicious traffic screening, the present disclosure also utilizes the deep pattern mining and outlier detection capability of clustering to further screen malicious traffic.

[0200] In some implementations of this disclosure, such as Figure 5 As shown, the specific process of filtering the second malicious traffic may include:

[0201] Enter network traffic data other than the first malicious traffic.

[0202] In some examples disclosed herein, all flows that appear in Malicious_Alert_List can be removed from ProcessedFeatureSet. k This is used to obtain the non-malicious traffic feature set, BenignFeatureSet. In some examples of this disclosure, all the aforementioned other network traffic data is initially marked as "unaccessed." This marking can be used to determine whether all the aforementioned other network traffic has been grouped into traffic clusters or marked as anomalous noise points.

[0203] Considering that DBSCAN is sensitive to feature scale and that standardization is key to DBSCAN's success, a set of numerical features that best represent the differences in popular behavior is selected from the ProcessedFeatureSet to suit the DBSCAN algorithm.

[0204] StandardizedBenignData

[0205] ={session_duration(flow k ),total_packets(flow k ),total_bytes(flow k ),avg_pkt_size(flow k ),pkt_size_stddev(flow k ),avg_interarrival_time(flow k ),interarrival_time_stddev(flow k ),avg_RTT(flow k ),avg_WindowSize(flow k )},in StandardizedBenignData has one row per column, and each column contains one of the selected features mentioned above (a total of 9 features, denoted as D=9).

[0206] In addition, it is necessary to determine the neighborhood radius eps and the minimum number of neighbors min_samples of the core point for the DBSCAN algorithm.

[0207] In some examples of the present disclosure, a value between min_samples ≥ D + 1 or min_samples = 2 * D can be chosen according to a dimension-based heuristic. min_samples represents the expectation that there are at least min_samples neighbors (including itself) around a core point. Then a neighborhood radius eps is chosen using k-distance plot: first k specifies the distance to the kth nearest neighbor of each point to be computed, take k = min_samples, for each stream in StandardizedBenignData, compute its Euclidean distance to its kth nearest neighbor, collect all these computed k-distance values, sort all k-distance values from large to small, plot the graph, Y axis is k-distance value, X axis is data point index, observe the graph, find the point where the curve slope changes most significantly, the Y axis value (k-distance) corresponding to this inflection point is a good candidate for eps. eps and min_samples can be fine-tuned in the following steps until a satisfactory clustering structure is obtained. For example, if there are too many noise points, eps may need to be increased or min_samples may need to be decreased; if there are almost no noise points and the clusters are too large, eps may need to be decreased.

[0208] Then, based on the neighborhood radius eps and the minimum number of neighbors min_samples of the core point, each stream in StandardizedBenignData is assigned to a cluster or marked as noise. The specific steps are as follows:

[0209] Randomly select a stream P from the unvisited streams, mark the stream as "visited": Visited(P) = False. Find the eps-neighborhood of P, i.e. all streams whose distance from P is less than or equal to eps: N eps (P) = {flow k | d(P, flow k ) ≤ eps}, where d(P, flow k ) is the distance between P and flow k .

[0210] If the number of streams in the neighborhood of P (including P itself) is greater than or equal to min_samples, P is considered a core point and a new cluster C is created: C = {P}, add all neighbors of P to the list to be expanded: Q = N eps (P).

[0211] (4) Traverse each point Q_point in the list Q to be expanded, if Q_point is not visited, mark Q_point as "visited": Visited(Q_point) = True; find the neighborhood of Q_point: N eps (Q_point) = {flow k |d(Q_point,flow k )≤eps}, if Q_point is a core point, add its neighbors to the list Q to be expanded: If Q_point does not belong to any cluster, add it to the current cluster C:

[0212] If the number of neighborhood flows of P is less than min_samples, mark it as a noise point:

[0213] Finally output a signature list, one-to-one corresponding to the rows of StandardizedBenignData. Each label represents the cluster ID (0, 1, 2,...) of the flow or is marked as a noise point (usually -1).

[0214] According to the feature portraits of the above traffic clusters and abnormal noise points, malicious traffic is screened. The traffic feature set BenignFeatureSet is grouped according to the clustering results (cluster labels), and each cluster represents a group of similar traffic features. The statistical information of the features is calculated for the traffic in each cluster, especially the core features such as the number of flows, mean, standard deviation, and other descriptive statistics. The feature statistics are also performed on the noise points (label -1) to find the features that are abnormal or different from normal traffic. The differences between different clusters are compared to identify the core features that cause the cluster grouping. The behavior of each cluster is described, with special attention to its traffic direction, byte size, packet size, and other features. Finally, the feature portrait of each cluster (including noise points) is output to describe the traffic pattern it represents and its potential security implications (such as normal traffic or malicious traffic).

[0215] Step S107 realizes that after determining the second malicious traffic or benign mode according to the feature portrait, the previously obtained malicious traffic alarm list can also be combined to further dynamically optimize the feedback security policy.

[0216] In some examples of the present disclosure, if some clusters are found to represent non-compliant behavior (such as a large amount of unauthorized P2P traffic, traffic connected to services that should not be accessed), corresponding network access control policies or bandwidth limitation policies can be formulated based on the characteristics of these clusters (such as IP address range, port, SNI). If a certain type of risk signal (such as a specific weak password suite, a suspicious SNI pattern) frequently appears in noise points or certain clusters, it can be considered to adjust the detection rules or thresholds of step 2, or add new detection logic. If a certain profiled cluster is considered to be completely benign, it can be considered to be added to the "white list" or to reduce its risk score in other detection systems.

[0217] Through the above steps, not only can key data support be provided for establishing differentiated security protection strategies and measures, but more importantly, various benign patterns and isolated traffic deviating from the main pattern existing in the network can be actively discovered, a refined baseline can be established, and potential suspicious threats can be discovered, and the security strategy can be continuously optimized.

[0218] In summary, the technical solution of the present disclosure discloses an encrypted network traffic analysis method based on multi-domain modeling, dynamic entropy driving and clustering technology, which aims to realize malicious activity identification, benign pattern discovery and abnormal traffic detection of encrypted network traffic.

[0219] As Figure 6As shown, the method first captures the original network traffic, filters and reconstructs it into TCP flows, and extracts detailed statistical, timing, state and encryption-related features from multiple levels such as flow, packet, TCP protocol and TLS handshake, standardizes and encodes to obtain the processed feature dataset. Then, the method maps the features to six semantic domains of network protocol, timing, behavior, statistics, encryption process and predefined malicious patterns, and discretizes the feature values into states; based on this multi-domain model, the Shannon entropy of each domain state distribution is calculated within the time window and dynamically weighted and aggregated, and the dynamic change rate and complexity of the global entropy are quantified by analyzing the Hessian matrix norm to generate an anomaly score to identify malicious traffic with sudden changes in behavior patterns. Next, the DBSCAN clustering algorithm is used to deeply mine stable benign patterns from unlabeled traffic and separate statistical outliers, achieving a synergistic enhancement of encrypted traffic anomaly detection and pattern understanding. Finally, the system outputs a malicious traffic alert list, a benign pattern cluster profile and an abnormal noise point set for further analysis. By combining the real-time anomaly detection capability of dynamic entropy with the deep pattern mining and outlier discovery capability of clustering, the present disclosure can effectively analyze encrypted traffic without decryption, and through continuous analysis and utilization of the identified malicious features, benign pattern profiles and abnormal noise points, the system can form a closed loop of learning and improvement, continuously optimizing the parameters and rules of the detection model, significantly improving the accuracy of malicious activity recognition, the depth of understanding of network behavior patterns, and providing data support for security policy optimization, achieving efficient and accurate monitoring of modern encrypted network environments.

[0220] It can be understood that the method can be executed by any device, equipment, platform, device cluster with computing and processing capability.

[0221] It should be noted that the method of one or more embodiments of the present disclosure can be executed by a single device, such as a computer or a server, etc. The method of the present embodiment can also be applied to a distributed scenario, and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only execute one or more steps in the method of one or more embodiments of the present disclosure, and the multiple devices will interact with each other to complete the method.

[0222] It should be noted that the above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in an order different than the order in the embodiments, and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0223] Based on the same inventive concept, the disclosure also provides a network malicious traffic detection and analysis device corresponding to the method of any of the above embodiments. As shown in Figure 2 The device includes:

[0224] The data acquisition module 11 is configured to acquire network traffic data, wherein the network traffic data includes encrypted network traffic based on the TCP protocol;

[0225] The feature extraction module 12 is configured to perform multi-dimensional feature extraction on the network traffic data and divide the extracted multi-dimensional features into multiple semantic domains;

[0226] The first calculation module 13 is configured to analyze the state distribution of each semantic domain within a preset time window, and obtain the Hessian matrix of the global entropy of each semantic domain according to the dynamic entropy of each semantic domain;

[0227] The first screening module 14 is configured to screen the first malicious traffic data in the network traffic data according to the Hessian matrix, wherein the first malicious traffic represents malicious traffic data with sudden changes in behavior patterns;

[0228] The second calculation module 15 is configured to apply the DBSCAN algorithm to cluster other network traffic data in the network traffic data except the malicious traffic data, to obtain at least one traffic cluster and an abnormal noise point;

[0229] The second screening module 16 is configured to screen the second malicious traffic data in the network traffic data from the traffic cluster and the abnormal noise point according to the feature portraits of the traffic cluster and the abnormal noise point, wherein the second malicious traffic data represents potential malicious traffic that is statistically outlying;

[0230] The strategy optimization module 17 is configured to adjust the traffic management strategy and / or the screening rules of the first malicious traffic data according to the feature portraits of the traffic cluster and the abnormal noise point.

[0231] Optionally, the 12 feature extraction module is specifically configured to:

[0232] Divide the multi-dimensional features of the network traffic data into multiple domains of network protocols, time sequences, traffic behaviors, traffic features, and encryption processes according to semantics.

[0233] Optionally, the 13 first calculation module is specifically configured to:

[0234] Discretize the multi-dimensional features of the network traffic data into intra-domain states;

[0235] determine whether the network traffic data activates a malicious behavior pattern in a malicious behavior pattern domain according to the in-domain state of the semantic domain, the malicious behavior pattern including at least one of a suspicious beacon pattern, a scanning probe pattern, an abnormal data exfiltration pattern, an abnormal payload download pattern, an encrypted negotiation pattern, and an unexpected behavior deviation pattern;

[0236] generate the in-domain state of the malicious behavior pattern domain according to the activation state of the malicious behavior pattern;

[0237] obtain a state distribution Shannon entropy of each of the semantic domain and the malicious behavior pattern domain according to the in-domain state of each of the semantic domain and the malicious behavior pattern domain within a preset time window;

[0238] weight and aggregate the state distribution Shannon entropy according to the importance of each of the semantic domain and the malicious behavior pattern domain to obtain a global entropy sequence, elements in the global entropy sequence being arranged in time sequence;

[0239] obtain the Hessian matrix according to the global entropy sequence.

[0240] Optionally, the first screening module 14 is specifically configured to:

[0241] obtain a Frobenius norm of the Hessian matrix according to the Hessian matrix, the Hessian matrix and the Frobenius norm representing the change rate and complexity of the state distribution Shannon entropy of each semantic domain;

[0242] obtain an anomaly score according to the Hessian matrix and the Frobenius norm, the anomaly score representing the change degree of the state distribution Shannon entropy of each semantic domain within a current time window;

[0243] determine that there is malicious traffic in the network traffic data in response to determining that the anomaly score is greater than or equal to a preset threshold;

[0244] analyze the network traffic data to determine first malicious traffic data.

[0245] Optionally, the second screening module 16 is specifically configured to:

[0246] perform feature statistics on the traffic cluster and the abnormal noise point;

[0247] obtain a first feature portrait of the traffic cluster and a second feature portrait of the abnormal noise point according to the features of the traffic cluster and the features of the abnormal noise point, the feature portrait being used to describe a representative traffic pattern and potential security implications;

[0248] According to the first feature image and the second feature image, the second malicious traffic data in the network traffic data is screened.

[0249] Optionally, the policy optimization module 17 is specifically configured to:

[0250] In response to any of the first feature images indicating that the traffic cluster has a violation behavior, a traffic management policy is formulated according to the first feature image, and the traffic management policy includes at least one of a network access control policy and a bandwidth limitation policy.

[0251] In response to the same type of risk signal appearing in multiple traffic clusters or abnormal noise points, the filtering rule of the first malicious traffic data is adjusted according to the type of the risk signal.

[0252] Optionally, the feature extraction module 12 is specifically configured to:

[0253] The features are extracted from multiple levels of the network traffic data, including flow, packet, TCP protocol, and TLS handshake process.

[0254] For the convenience of description, the above apparatus is described in various modules according to functions. Of course, the functions of each module can be implemented in one or more software and / or hardware when implementing one or more embodiments of the present disclosure.

[0255] The apparatus of the above embodiments is used to implement the corresponding method in the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here.

[0256] Figure 7 A more specific hardware structure of an electronic device is shown, which can include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 for communication within the device.

[0257] The processor 1010 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0258] The memory 1020 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present disclosure are implemented by software or firmware, the related program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0259] The input / output interface 1030 is configured to connect an input / output module to realize information input and output. The input / output module can be configured in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0260] The communication interface 1040 is configured to connect a communication module (not shown in the figure) to realize the communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as USB, network cable, etc.) or through a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0261] The bus 1050 includes a channel to transmit information between various components (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0262] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only contain the components necessary to implement the solutions of the embodiments of the present disclosure, and does not have to contain all the components shown in the figure.

[0263] The electronic device of the above embodiments is used to implement the corresponding methods in the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here again.

[0264] The computer readable media of the present embodiments includes permanent and non-permanent, removable and non-removable media can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device.

[0265] Those skilled in the art will understand that the above discussion of any of the embodiments is merely exemplary and is not intended to be limiting of the scope of the disclosure, including claims, to these examples; the above embodiments or technical features between different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of one or more embodiments of the disclosure as described above, which are not provided in detail for the sake of brevity.

[0266] In addition, in order to simplify the description and discussion, and so as not to make one or more embodiments of the disclosure difficult to understand, the known power / ground connections of integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. In addition, the devices can be shown in the form of block diagrams in order to avoid making one or more embodiments of the disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented one or more embodiments of the disclosure (i.e. these details should be fully within the understanding of those skilled in the art). Where specific details (e.g. circuits) are set forth in order to describe exemplary embodiments of the disclosure, it will be apparent to those skilled in the art that the one or more embodiments of the disclosure can be implemented without these specific details or with variations on these specific details. Therefore, these descriptions should be considered illustrative rather than limiting.

[0267] Although the disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications and variations will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g. dynamic RAM (DRAM)) can use the embodiments discussed.

[0268] One or more embodiments of the present disclosure are intended to cover all such alternatives, modifications, and variations as come within the scope of the broadest possible interpretation of the appended claims. Accordingly, any and all such alternations, modifications, equivalents, improvements and the like are intended to be encompassed by the present disclosure.

Claims

1. A network malicious traffic detection analysis method, characterized by, The method comprises: obtaining network traffic data, the network traffic data comprising encrypted network traffic based on a TCP protocol; performing multi-dimensional feature extraction on the network traffic data, and dividing the extracted multi-dimensional features into a plurality of semantic domains; analyzing a state distribution of each of the semantic domains within a preset time window, and obtaining a Hessian matrix of a global entropy of each of the semantic domains according to a dynamic entropy of each of the semantic domains; screening first malicious traffic data in the network traffic data according to the Hessian matrix, the first malicious traffic data representing malicious traffic data with a behavior pattern mutation; applying a DBSCAN algorithm to cluster other network traffic data in the network traffic data except the malicious traffic data, to obtain at least one traffic cluster and an abnormal noise point; screening second malicious traffic data in the network traffic data from the traffic cluster and the abnormal noise point according to a feature portrait of the traffic cluster and the abnormal noise point, the second malicious traffic data representing potential malicious traffic with statistical outliers; adjusting a traffic management strategy and / or a screening rule of the first malicious traffic data according to the feature portrait of the traffic cluster and the abnormal noise point.

2. The method of claim 1, wherein, The multi-dimensional features of the network traffic data are divided into a plurality of domains of network protocols, time sequences, traffic behaviors, traffic features, and encryption processes according to semantics.

3. The method of claim 2, wherein, The analyzing of the state distribution of each of the semantic domains within the preset time window comprises: discretizing the multi-dimensional features of the network traffic data into an in-domain state; judging whether the network traffic data activates a malicious behavior pattern in a malicious behavior pattern domain according to the in-domain state of the semantic domain, the malicious behavior pattern comprising at least one of a suspicious beacon pattern, a scanning and investigation pattern, an abnormal data exfiltration pattern, an abnormal payload download pattern, an encryption negotiation pattern, and an unexpected behavior deviation pattern; generating an in-domain state of the malicious behavior pattern domain according to an activation state of the malicious behavior pattern; obtaining a state distribution Shannon entropy of each of the semantic domains and the malicious behavior pattern domain according to the in-domain states of each of the semantic domains and the malicious behavior pattern domain within the preset time window; weighting and aggregating the state distribution Shannon entropy according to the importance of each of the semantic domains and the malicious behavior pattern domain, to obtain a global entropy sequence, elements in the global entropy sequence being arranged in a time sequence; obtaining the Hessian matrix according to the global entropy sequence.

4. The method of claim 3, wherein, The screening of the first malicious traffic data in the network traffic data according to the Hessian matrix comprises: obtaining a Frobenius norm of the Hessian matrix according to the Hessian matrix, the Hessian matrix and the Frobenius norm representing a change rate and a complexity of the state distribution Shannon entropy of each semantic domain; According to the Hessian matrix and the Frobenius norm, an anomaly score is obtained, the anomaly score representing a degree of change of state distribution Shannon entropy of each semantic domain in a current time window; In response to determining that the anomaly score is greater than or equal to a preset threshold, it is determined that malicious traffic exists in the network traffic data; The network traffic data is analyzed to determine first malicious traffic data.

5. The method of claim 1, wherein, The second malicious traffic data in the network traffic data is screened from the traffic cluster and the abnormal noise point according to the feature portraits of the traffic cluster and the abnormal noise point, including: The features of the traffic cluster and the abnormal noise point are counted; According to the features of the traffic cluster and the features of the abnormal noise point, a first feature portrait of the traffic cluster and a second feature portrait of the abnormal noise point are obtained, the feature portraits being used to describe representative traffic patterns and potential security implications; According to the first feature portrait and the second feature portrait, the second malicious traffic data in the network traffic data is screened.

6. The method of claim 5, wherein, The feature portraits of the traffic cluster and the abnormal noise point are adjusted, including: In response to any of the first feature portraits indicating that the traffic cluster has a violation, a traffic management strategy is formulated according to the first feature portrait, the traffic management strategy including at least one of a network access control strategy and a bandwidth restriction strategy; In response to the same type of risk signal appearing in multiple traffic clusters or abnormal noise points, the screening rules of the first malicious traffic data are adjusted according to the type of the risk signal.

7. The method of claim 1, wherein, Feature extraction is performed on the network traffic data from multiple levels in the flow, packet, TCP protocol and TLS handshake process.

8. A network malicious traffic detection analysis apparatus, characterized by, Including: A data acquisition module configured to acquire network traffic data, the network traffic data including encrypted network traffic based on a TCP protocol; A feature extraction module configured to perform multi-dimensional feature extraction on the network traffic data, and divide the extracted multi-dimensional features into multiple semantic domains; A first calculation module configured to analyze the state distribution of each semantic domain in a preset time window, and obtain a Hessian matrix of global entropy of each semantic domain according to the dynamic entropy of each semantic domain; A first screening module configured to screen first malicious traffic data in the network traffic data according to the Hessian matrix, the first malicious traffic representing malicious traffic data with a sudden change in behavior pattern; A second calculation module configured to apply a DBSCAN algorithm to cluster other network traffic data in the network traffic data except the malicious traffic data, to obtain at least one traffic cluster and abnormal noise point; A second screening module configured to screen second malicious traffic data in the network traffic data from the traffic cluster and the abnormal noise point according to feature portraits of the traffic cluster and the abnormal noise point, the second malicious traffic data representing potential malicious traffic that is statistically outlying. The policy optimization module is configured to adjust a traffic management policy and / or a screening rule of the first malicious traffic data according to the feature portraits of the traffic cluster and the abnormal noise points.

9. The apparatus of claim 8, wherein, The feature extraction module is specifically configured to: divide the multi-dimensional features of the network traffic data into a plurality of domains of network protocols, time sequences, traffic behaviors, traffic features, and encryption processes according to semantics.

10. The apparatus of claim 9, wherein, The first calculation module is specifically configured to: discretize the multi-dimensional features of the network traffic data into intra-domain states; determine whether the network traffic data activates a malicious behavior mode in a malicious behavior mode domain according to the intra-domain states of the semantic domains, the malicious behavior mode including at least one of a suspicious beacon mode, a scanning and investigation mode, an abnormal data export mode, an abnormal payload download mode, an encryption negotiation mode, and an unexpected behavior deviation mode; generate an intra-domain state of the malicious behavior mode domain according to an activation state of the malicious behavior mode; obtain a state distribution Shannon entropy of each of the semantic domains and the malicious behavior mode domain according to the intra-domain states of each of the semantic domains and the malicious behavior mode domain within a preset time window; weight and aggregate the state distribution Shannon entropies according to importance of each of the semantic domains and the malicious behavior mode domain to obtain a global entropy sequence, elements in the global entropy sequence being arranged in time sequence; obtain the Hessian matrix according to the global entropy sequence.

11. The apparatus of claim 10, wherein, The first screening module is specifically configured to: obtain a Frobenius norm of the Hessian matrix according to the Hessian matrix, the Hessian matrix and the Frobenius norm representing a change rate and a complexity of the state distribution Shannon entropies of each semantic domain; obtain an anomaly score according to the Hessian matrix and the Frobenius norm, the anomaly score representing a change degree of the state distribution Shannon entropies of each semantic domain within a current time window; determine that malicious traffic exists in the network traffic data in response to determining that the anomaly score is greater than or equal to a preset threshold; analyze the network traffic data to determine first malicious traffic data.

12. The apparatus of claim 8, wherein, The second screening module is specifically configured to: perform feature statistics on the traffic cluster and the abnormal noise points; obtain a first feature portrait of the traffic cluster and a second feature portrait of the abnormal noise points according to features of the traffic cluster and features of the abnormal noise points, the feature portraits being used to describe a representative traffic mode and a potential security implication; screen second malicious traffic data in the network traffic data according to the first feature portrait and the second feature portrait.

13. The apparatus of claim 12, wherein, The policy optimization module 17 is specifically configured to: formulate a traffic management policy according to the first feature portrait in response to any of the first feature portraits indicating that the traffic cluster has a violation, the traffic management policy including at least one of a network access control policy and a bandwidth limitation policy; In response to the same type of risk signal appearing in multiple of the traffic clusters or the abnormal noise points, adjusting a screening rule of the first malicious traffic data according to the type of the risk signal.

14. The apparatus of claim 8, wherein, The feature extraction module is specifically configured to: Extracting features from multiple levels in the network traffic data, including flow, packet, TCP protocol, and TLS handshake process.

15. An electronic device comprising a memory, a processor, and a computer program stored on the memory and run by the processor, characterized in that, The processor implements the method of any one of claims 1 to 7 when executing the computer program.

16. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions for causing the computer to execute the method of any one of claims 1 to 7.

Images