A multi-stage detection method and system for malicious VPN traffic

Abstract

The present application relates to the technical field of network security, and particularly relates to a multi-stage detection method and system for malicious VPN traffic, which comprises the following steps: loading a malicious VPN IP address library and a domain name library and converting them into a required dictionary tree structure; configuring an uplink and downlink traffic mirroring strategy to capture network traffic in real time; extracting information from the traffic; inputting destination IP and domain name features into the dictionary tree for fast matching; triggering an alarm and solidifying traffic link information if the matching is successful; if there is no hit, enabling a JA3 and JA4 joint fingerprint identification mechanism to determine whether it is VPN traffic through similarity calculation, and adding newly discovered features to the feature library for dynamic updating; generating a work order automatically and notifying the person in charge after detecting malicious traffic, and starting an SLA timer to track the disposal process. The present application can achieve microsecond-level interception of known malicious VPN traffic, and effectively identify unknown variants of VPN tunnels disguised as HTTPS.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a multi-stage detection method and system for malicious VPN traffic. Background Technology

[0002] In a corporate network environment, employees' unauthorized use of VPNs to access overseas networks not only violates relevant laws and regulations but also poses a serious threat to corporate network security. From a corporate security perspective, employees using malicious VPN software is equivalent to tearing open an unmonitored gap in the corporate security boundary. Trade secrets, customer data, and core technical information may be leaked through this channel at any time, leading to serious economic losses and legal disputes.

[0003] Currently, enterprise security teams face enormous challenges when investigating VPN traffic. Modern enterprise networks generate terabytes of outbound traffic daily, with tens of thousands of endpoints simultaneously communicating with massive amounts of internet IPs. Security personnel need to process the vast amounts of log data generated by firewalls, proxy servers, and other devices; even completing basic classification and sorting is overwhelming. A more critical issue is that the encryption protocols (such as TLS and IPSec) upon which VPN traffic relies are highly similar in characteristics to normal business traffic such as ordinary HTTPS web browsing and email transmission at the packet level. Purely manual investigation without deep packet inspection and protocol signature analysis is almost impossible to distinguish whether an outbound connection is a regular office access or a cross-border connection disguised as a VPN tunnel.

[0004] Meanwhile, circumvention activities often occur during periods of weak supervision, such as late at night and weekends. Manual log analysis can only be conducted after the fact, and by the time an anomaly is detected, the connection has already ended. Even if a suspicious IP address and port are found, it is impossible to confirm whether the IP is shared by multiple people, whether the behavior is occasional or long-term, or whether there are other hidden tunnels on the same terminal. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-stage detection method and system for malicious VPN traffic, in order to solve the technical problem that existing technologies have difficulty in effectively detecting large amounts of malicious VPN traffic.

[0006] One aspect of the present invention provides a multi-stage detection method for malicious VPN traffic, comprising: S1. Load the malicious VPN IP address database and malicious VPN domain name database into the detection system memory, and perform data structuring preprocessing to convert the IP address and domain name information into the trie structure required by the DFA algorithm; S2. Configure uplink and downlink bidirectional traffic mirroring strategy to bypass network data packets to the detection system via port mirroring; S3. Read the mirror traffic data in real time through the packet capture interface, perform multi-layer protocol stack decoding, and extract application layer payload data; S4. Extract communication quintuple information, traffic content characteristics, and IP address location information from the parsed traffic; S5. Input the extracted destination IP address and domain name features into the pre-built DFA trie for fast matching: S5.1. If the DFA match is successful, an alarm process is triggered, a security event object is generated, and complete traffic data within the time window before and after the malicious session is saved. S5.2. If DFA matching fails, the JA3 and JA4 joint fingerprinting mechanism will be enabled to determine VPN traffic. S6. Automatically generate structured security response work orders and send alarm information to the security manager via email, SMS, and enterprise IM channels; S7. Track the handling process, start the SLA timer, and form a closed-loop process of detection, alarm, notification, and handling.

[0007] In some embodiments, the data structuring preprocessing includes converting the set of IP addresses and domain names into a Trie tree structure, and completing the construction of state nodes and the generation of jump tables.

[0008] In some embodiments, a segmented Trie tree structure is used for IP addresses, which are split into four bytes in dotted decimal format for storage; a character-level Trie tree structure is used for domain names, which stores each character in the domain name string as a tree node.

[0009] In some embodiments, the configuration of the uplink and downlink bidirectional traffic mirroring policy adopts the SPAN port mirroring policy.

[0010] In some embodiments, the real-time reading of mirrored traffic data via the packet capture interface uses PF_RING, DPDK, or libpcap packet capture interfaces for traffic acquisition. In some embodiments, the traffic content features include the SNI field in the TLS handshake message, the Host field, User-Agent field in the HTTP protocol, the certificate serial number, and certificate authority information; the IP address location information is obtained by querying the GeoIP geographic information database, including the country, province, city, and ISP operator information of the IP address.

[0011] In some embodiments, if the DFA match is successful, it is directly determined to be known malicious VPN traffic, and a packet capture file in pcap format or a structured JSON log is generated to solidify the traffic communication link information and record the source identifier, destination identifier, intermediate network nodes, session duration, packet sequence number and byte stream statistics.

[0012] In some embodiments, the JA3 algorithm extracts the TLS version number, cipher suite list and its order, extended field list, elliptic curve type and dot format parameters from the TLS ClientHello message, and calculates and generates a hash fingerprint. The JA4 algorithm further incorporates HTTP / 2 frame structure features, QUIC protocol features, certificate chain information, and timestamp dimensions based on the JA3 algorithm to construct a highly distinguishable protocol fingerprint. By storing JA3 or JA4 fingerprints of known malicious VPNs in a pre-built VPN client fingerprint database, the Jaccard similarity between real-time traffic fingerprints and fingerprints in the database is calculated. If the similarity exceeds a preset threshold, it is determined to be VPN traffic.

[0013] In some embodiments, the method further includes adding the domain name information and IP address corresponding to the traffic that successfully matches the JA3 or JA4 fingerprint to the malicious VPN domain name database and IP database, and reinitializing the DFA dictionary tree.

[0014] Another aspect of the present invention provides a multi-stage detection system for malicious VPN traffic, comprising: a signature database loading module, a traffic mirroring configuration module, a traffic capture and parsing module, a metadata extraction module, a DFA matching module, a JA3 / JA4 fingerprint detection module, an alarm and link solidification module, a work order and notification module, and a closed-loop management module for handling; wherein: The feature library loading module is used to load the malicious VPN IP address library and the malicious VPN domain name library, and convert the IP address and domain name information into a trie structure required by the DFA algorithm; the feature library loading module is specifically used to convert the set of IP addresses and domain names into a Trie tree structure, and complete the construction of state nodes and the generation of jump tables; The traffic mirroring configuration module is used to configure uplink and downlink bidirectional traffic mirroring policies at network nodes, bypassing network data packets to the detection system; the traffic mirroring configuration module adopts the SPAN port mirroring policy; The traffic capture and parsing module is used to read mirrored traffic data in real time through the packet capture interface, perform multi-layer protocol stack decoding, and extract application layer payload; the traffic capture and parsing module uses PF_RING, DPDK or libpcap packet capture interfaces for traffic collection; The metadata extraction module is used to extract communication quintuples, traffic content features, and IP address location information. The traffic content features include the SNI field in the TLS handshake message, the Host field, User-Agent field, certificate serial number, and certificate authority information in the HTTP protocol. The IP address location information is obtained by querying the GeoIP geographic information database or the ASN routing database, and includes the country, province, city, and ISP operator information of the IP address. The DFA matching module is used to input the extracted traffic features into the DFA trie for fast matching; if the match is successful, the alarm process is triggered directly. The JA3 / JA4 fingerprint detection module is used to perform JA3 / JA4 fingerprint identification on traffic that fails to match the DFA to determine whether it is VPN traffic. The JA3 / JA4 fingerprint detection module also includes a pre-built VPN client / server fingerprint database, which is used to store the JA3 / JA4 fingerprints of known malicious VPNs and to determine whether the traffic is VPN traffic through Jaccard similarity calculation or threshold matching. The JA3 / JA4 fingerprint detection module is also used to add the domain name information and IP address corresponding to the traffic with successfully matched fingerprints to the malicious VPN domain name database and IP database, triggering the re-initialization of the DFA trie. The alarm and link solidification module is used to trigger alarms and save traffic communication link information when malicious traffic is detected, generate packet capture files in pcap format or structured JSON logs, and record source identifier, destination identifier, intermediate network nodes, session duration, data packet sequence number and byte stream statistics. The work order and notification module is used to generate safety handling work orders and send alarm information via email, SMS, and enterprise IM channels; the work order and notification module connects to third-party platforms through a Webhook interface to push card messages and supports one-click jump to the work order details page; The closed-loop management module is used to track the handling process, start the SLA timer and include an overdue escalation mechanism. When a work order is not processed within the preset time limit, it is automatically escalated to the next higher level of management for handling supervision.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The DFA algorithm is used for first-level feature matching, which enables microsecond-level interception of known malicious VPN traffic characteristics, significantly reducing the computational load of subsequent complex algorithms and improving overall detection efficiency; 2. Combining JA3 and JA4 fingerprinting algorithms, it can effectively identify unknown, variant, or self-developed VPN tunnels disguised as ordinary HTTPS traffic, solving the technical problem of difficult identification of encrypted traffic. It realizes full-process automation from traffic detection, alarm triggering, work order generation to multi-channel notification, and can complete the discovery, notification, and initial response of security incidents without manual intervention. 3. Automatically save complete traffic data within the time window before and after malicious sessions, generating unalterable pcap files or structured logs, providing a complete chain of evidence for security auditing, source tracing and countermeasures, and judicial evidence collection; 4. By using a multi-channel parallel notification mechanism, ensure that alarm information can reach the security manager as soon as possible, shortening the response time for security incidents. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Fig. 1 This is a flowchart of the multi-stage detection method for malicious VPN traffic of the present invention; Fig. 2 This is a diagram of the multi-stage detection system architecture for malicious VPN traffic according to the present invention. Detailed Implementation

[0018] The following will be based on embodiments of the present invention. Figs. 1-2 The technical solutions in the embodiments of the present invention will be clearly and completely described together. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0019] Example 1 This embodiment details a multi-stage detection system for malicious VPN traffic.

[0020] The multi-stage detection system for malicious VPN traffic includes a signature database loading module, a traffic mirroring configuration module, a traffic capture and parsing module, a metadata extraction module, a DFA matching module, a JA3 / JA4 fingerprint detection module, an alarm and link solidification module, a work order and notification module, a closed-loop management module for handling, a malicious VPN IP database, a malicious VPN domain database, a VPN client / server fingerprint database, a GeoIP geographic information database, an email notification module, an SMS notification module, and an enterprise IM notification module.

[0021] The feature library loading module loads the malicious VPN IP address database and malicious VPN domain name database into the detection system memory. It then performs data structuring preprocessing on these databases, converting the IP address and domain name information into a trie structure required by the DFA algorithm. This data structuring preprocessing includes converting the IP address and domain name set into a Trie tree or AC automaton structure, constructing state nodes and generating jump tables, achieving high-speed matching with O(N) time complexity.

[0022] In a specific example, the feature library loading module reads the list of malicious VPN IP addresses and malicious VPN domain names stored in the disk file, parses each IP address or domain name record, and constructs the corresponding Trie tree node. For IP addresses, a segmented Trie tree structure is adopted, splitting the IP address into four bytes in dotted decimal format, with each byte stored as a first-level node of the Trie tree. For domain names, a character-level Trie tree structure is adopted, storing each character in the domain name string as a tree node. Domain names with the same prefix share the prefix node, effectively reducing storage space usage. After completing the Trie tree construction, the feature library loading module generates a state transition table, which records the index of the next state node when each state node receives different input characters, supporting state transition queries with O(1) time complexity.

[0023] The traffic mirroring configuration module is used to configure uplink and downlink bidirectional traffic mirroring policies at core network switching nodes or security gateways. This allows network packets to be bypassed and routed to the detection system without loss via port mirroring. The module employs SPAN or RSPAN port mirroring policies to achieve zero-latency, zero-packet-loss traffic collection for existing network services.

[0024] In a specific example, the network administrator configures a SPAN source port on the core switch to copy all traffic from the internal network segment to be monitored to a designated SPAN destination port. The SPAN destination port is connected to the traffic capture network interface card (NIC) of the detection system, which receives the copied network packets through this NIC. During configuration, it is crucial to ensure that the bandwidth of the SPAN destination port is not less than the sum of the bandwidths of all SPAN source ports to avoid packet loss due to insufficient port bandwidth. For cross-switch traffic monitoring scenarios, RSPAN technology is used, transmitting the monitored traffic to the remote detection system via a specific VLAN. The bandwidth of the RSPAN VLAN must also meet the traffic bandwidth requirements. The traffic mirroring configuration module also supports configuring traffic filtering rules to mirror only traffic within a specific five-tuple range, reducing the processing load on the detection system.

[0025] The traffic capture and parsing module is used to read mirrored traffic data streams in real time through the packet capture interface, perform multi-layer protocol stack decoding, and extract application layer payload data. The traffic capture and parsing module uses the PF_RING, DPDK, or libpcap packet capture interface for traffic collection. In a specific example, after the traffic capture and parsing module starts, it first initializes the PF_RING ring buffer, setting the size of each packet slot to 2048 bytes, and the total buffer size to 512MB, which can hold approximately 260,000 packets. After initialization, the traffic capture and parsing module continuously receives packets from the network card through the PF_RING polling interface and stores the packets in the ring buffer. For each received packet, the traffic capture and parsing module calls the protocol parser to parse the packet layer by layer, extracting the Ethernet header, IP header, and TCP / UDP header information, and determining the upper-layer application layer protocol type based on the TCP / UDP port number. For HTTP traffic, the HTTP parser is invoked to extract the request line, request headers, and request body. For HTTPS traffic, the TLS parser is first invoked to extract the ClientHello and ServerHello information from the TLS handshake message, as well as the SNI extension fields, the list of supported cipher suites, elliptic curve parameters, etc. For DNS traffic, DNS query and response messages are parsed to extract the queried domain name and the resolved IP address. The traffic capture and parsing module encapsulates the parsed data into a structured traffic metadata object, which is then passed to the metadata extraction module for processing.

[0026] The metadata extraction module extracts communication 5-tuple information, traffic content characteristics, and IP address geolocation information from the parsed traffic. The communication 5-tuple information includes the source IP address, source port number, destination IP address, destination port number, and transport protocol type. Traffic content characteristics include the SNI field in TLS handshake messages, the Host field, User-Agent field, certificate sequence number, and certificate authority information in HTTP protocols. IP address geolocation information is obtained by querying the GeoIP geographic information database or the ASN routing database, including the country, province, city, and ISP information of the IP address.

[0027] In a specific example, after receiving the traffic metadata object from the traffic capture and parsing module, the metadata extraction module first extracts the five-tuple field of the object and encapsulates it into a FlowTuple object. For TLS traffic, the metadata extraction module further extracts the SNI extended field value from the TLS ClientHello message. If the SNI field does not exist or is empty, the domain name information is extracted from the HTTP Host header as a substitute. For HTTP traffic, the metadata extraction module extracts the URL path from the request line, the User-Agent field value from the request header, the Accept-Language field value, etc. For certificate information, after the TLS handshake reaches the ServerHello stage, the metadata extraction module extracts the certificate serial number and issuing authority name from the X.509 certificate. When extracting IP address geolocation information, the metadata extraction module first checks the local cache to see if the IP address already has cached geolocation information. If not, it calls the query interface of the GeoIP geographic information database, passing in the source IP address and destination IP address to perform the query, obtaining the corresponding country code, province name, city name, and ISP operator name, and caches the query results in memory for subsequent fast access. The metadata extraction module encapsulates all extracted metadata into a complete FlowMetadata object and passes it to the DFA matching module for first-level feature matching.

[0028] The DFA matching module is used to input the extracted traffic features into a pre-built DFA dictionary tree for fast matching; if the match is successful, it is directly identified as known malicious VPN traffic and an alarm process is triggered.

[0029] In a specific example, after receiving the FlowMetadata object, the DFA matching module extracts the destination IP address and domain name characteristics from the object as matching keywords. The DFA matching module first performs IP address matching, converting the destination IP address into a four-byte integer representation. Starting from the root node of the Trie tree, it sequentially performs state transitions according to the four bytes of the IP address, checking if the destination IP address exists in the malicious VPN IP database. If a complete destination IP address node is found in the Trie tree, it indicates that the IP address belongs to a known malicious VPN IP address, and the DFA matching module immediately returns a successful match result, along with the risk level label and associated threat intelligence information corresponding to the malicious IP. If the IP address matching fails, the DFA matching module continues with domain name matching, using the domain name characteristics as keywords to perform prefix matching in the Trie tree corresponding to the malicious VPN domain name database. Domain name matching uses the longest prefix matching strategy; if the domain name contained in the traffic is a subdomain of a malicious VPN domain name or is a complete match, it is considered a successful match. The matching process of the DFA matching module is implemented using a state transition table. The time complexity of the state transition is O(1), and the time complexity of the entire matching process is O(L), where L is the length of the IP address or domain name. In the actual network environment, L usually does not exceed 15 characters, so the matching time can be controlled at the microsecond level.

[0030] When the DFA matching module successfully matches, the alarm and link solidification module immediately triggers the alarm process, generates a high-risk security event object, and automatically associates the complete traffic data within the time window before and after the malicious session, generates a packet capture file in pcap format or a structured JSON log, solidifies the traffic communication link information, and records the source identifier, destination identifier, intermediate network nodes, session duration, packet sequence number and byte stream statistics.

[0031] In a specific example, the alarm and link solidification module first creates an AlertEvent object, sets the event level to high risk, the event type to malicious VPN traffic, and the event occurrence time to the current system time. It then populates the object with information extracted from the FlowMetadata object, including the source IP address, source port, destination IP address, destination port, protocol type, source IP address location, and destination IP address location. Simultaneously, the alarm and link solidification module initiates traffic backtracking, retrieving all data packets within a 60-second time window before and after the session from the circular buffer of the traffic capture and parsing module. These packets are then reassembled into a pcap file in chronological order and written to the forensic storage directory. The forensic pcap file is named "alert_timestamp_source IP_destination IP.pcap" for easy subsequent source tracing analysis. The alarm and link solidification module also generates a structured JSON log, recording complete information about the security event, including the alarm number, event type, risk level, attack source information, attack target information, traffic statistics, and associated threat intelligence tags. This JSON log is also written to the storage directory. After generating an alarm event, the alarm and link solidification module sends the alarm message to the work order and notification module for further processing.

[0032] If the DFA matching module fails to match, the JA3 / JA4 fingerprint detection module enables a joint JA3 and JA4 fingerprint recognition mechanism for encrypted traffic. The JA3 algorithm extracts the TLS version number, cipher suite list and its order, extended field list, elliptic curve type, and dot format parameters from the TLS ClientHello message to generate a hash fingerprint. The JA4 algorithm further incorporates HTTP / 2 frame structure features, QUIC protocol features, certificate chain information, and timestamp dimensions on top of the JA3 fingerprint to construct a highly distinguishable protocol fingerprint. The JA3 / JA4 fingerprint detection module stores the JA3 / JA4 fingerprints of known malicious VPNs in a pre-built VPN client / server fingerprint database. It calculates the Jaccard similarity or cosine similarity between the real-time traffic fingerprint and each fingerprint in the database. If the similarity exceeds a preset threshold, the traffic is identified as VPN traffic.

[0033] In a specific example, the JA3 / JA4 fingerprint detection module first determines whether the FlowMetadata object includes TLS handshake information. If it does not include TLS information, the fingerprint detection process is skipped, and the traffic is classified as non-VPN traffic. If TLS information is included, the JA3 / JA4 fingerprint detection module extracts the values ​​of various fields from the TLS ClientHello message, including the TLS version number, the list and order of supported cipher suites, the list of supported elliptic curves, the list of supported dotted curve formats, and the list of extensions. These fields are then concatenated according to a specific format and the MD5 hash algorithm is used to calculate the JA3 fingerprint value. For traffic supporting HTTP / 2, the JA3 / JA4 fingerprint detection module additionally extracts the setting parameters and window update parameters of the HTTP / 2 frame, concatenates them to the JA3 fingerprint string, and calculates the MD5 hash again to obtain the JA4 fingerprint value. The JA3 / JA4 fingerprint detection module then compares the calculated JA3 / JA4 fingerprint value with fingerprints in a pre-stored VPN client / server fingerprint database. Each record in the fingerprint database contains known malicious VPN fingerprint values, the VPN client or server type corresponding to that fingerprint, and the risk level, among other attributes. Fingerprint comparison employs the Jaccard similarity calculation method, which calculates the intersection and union of the fingerprint field set of real-time traffic with the field sets of each record in the fingerprint database to obtain a similarity value. If the similarity exceeds a preset threshold of 0.8, the traffic is determined to be VPN traffic. The JA3 / JA4 fingerprint detection module returns the successful fingerprint matching result to the upper-layer processing flow.

[0034] The JA3 / JA4 fingerprint detection module is also used to add the domain name information and IP address corresponding to the traffic that successfully matches the JA3 / JA4 fingerprint to the malicious VPN domain name database 11 and the malicious VPN IP database, triggering the feature database loading module to reinitialize the DFA dictionary tree and realize the dynamic update of the feature database.

[0035] In a specific example, when the JA3 / JA4 fingerprint detection module determines that a certain traffic is VPN traffic, it extracts the destination IP address and domain name information of that traffic from the FlowMetadata object. It then calls the signature database update interface to write the new malicious IP address and malicious domain name into the storage files of the malicious VPN IP database and malicious VPN domain name database, respectively. When the signature database loading module detects a change in the signature database file, it automatically rereads the updated signature database file, reconstructs the Trie tree structure and state transition table, ensuring that newly discovered malicious VPN features can be promptly identified in subsequent DFA matching. This dynamic update mechanism allows the detection system to continuously learn and accumulate newly emerging malicious VPN features, improving its overall detection capabilities.

[0036] The work order and notification module automatically generates structured security handling work orders and sends alarm information to security managers through three channels: email notification, SMS notification, and enterprise IM notification. The work order and notification module connects to DingTalk, WeChat Work, or Lark platforms via a Webhook interface to push card messages and supports one-click jump to the work order details page. In a specific embodiment, after receiving the AlertEvent object from the alarm and link solidification module, the work order and notification module first creates a security handling work order (WorkOrder), sets the work order number to a system timestamp plus a random string, sets the work order status to pending, and determines the work order priority based on the risk level of the alarm event. The work order content includes detailed information about the alarm event, the path to the associated forensic files, and suggested handling measures. After the work order is created, the work order and notification module calls the email sending interface to send an alarm email to the preset security manager's email address via the email notification module. The email content includes an alarm summary and a link to the work order details page. Simultaneously, the work order and notification module calls the SMS gateway interface to send an alarm SMS to the preset security manager's mobile phone number via the SMS notification module. The SMS content includes a brief alarm message and the work order number. For enterprise IM notifications, the ticket and notification modules select the appropriate Webhook interface to interface with DingTalk, WeChat Work, or Lark based on the configuration. They construct card message content including alarm details and send it to the Webhook address via an HTTP POST request. The card message includes key information such as alarm level, alarm type, occurrence time, source IP, and destination IP, and embeds a URL link to the ticket details page, allowing security personnel to click the link to jump directly to the ticket system to view and process the ticket.

[0037] The closed-loop management module is used to track the handling process, start the SLA timer and include an overdue escalation mechanism. When a work order is not processed within the preset time limit, it is automatically escalated to the next higher level of management for handling supervision.

[0038] In a specific example, the closed-loop management module starts an SLA timer after a work order is created, setting a preset processing time limit for the work order, with a default time limit of 4 hours. The module checks the processing status of all pending work orders every minute. For work orders that have not been processed within the preset time limit, a timeout escalation mechanism is triggered. This mechanism first obtains the information of the current handler's superior, constructs an escalation notification, and sends an alarm notification to the escalated manager through the work order and notification module. Notification methods include email, SMS, and enterprise instant messaging. The escalation notification includes the original work order information, timeout duration, and escalation reason. The closed-loop management module also records the work order's status change history, including the work order creation time, first processing time, work order closure time, and closure reason, for subsequent statistical analysis and compliance auditing. Once the security manager processes the alarm work order in the work order system and marks it as processed, the closed-loop management module ends the SLA timer for that work order, calculates the actual processing time, and updates the work order's processing status. If the security manager marks the work order as a false alarm, the closed-loop management module will mark the work order as a false alarm and remove the corresponding features from the malicious VPN IP database and malicious VPN domain database, triggering a reload of the feature database.

[0039] Example 2 In practical applications, the system first loads predefined malicious VPN IP addresses and domain names into memory using a feature library loading module and constructs an efficient Trie tree index structure. The system configures a traffic mirroring policy at the core network switching node, replicating bidirectional traffic to the packet capture network interface card of the detection system. The traffic capture and parsing module continuously captures network packets through the PF_RING interface and performs multi-layer protocol parsing to extract key application layer information. The metadata extraction module extracts rich features from the parsed traffic, including five-tuple information, TLS fingerprints, HTTP header information, and IP address attribution. The DFA matching module uses the Trie tree index to perform high-speed matching of destination IPs and domain names, achieving microsecond-level detection of known malicious VPN traffic. When a DFA match fails, the JA3 / JA4 fingerprint detection module uses TLS fingerprinting technology to detect unknown or variant VPN tunnels. Upon detecting malicious traffic, the alarm and link solidification module automatically saves evidence data and logs, the work order and notification module sends alarms to the security manager through multiple channels, and the closed-loop management module tracks the entire handling process and automatically escalates upon timeout, forming a complete automated security operation closed loop of detection, alarm, notification, handling, and review.

[0040] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the scope of the invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0041] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multi-stage detection method and system for malicious VPN traffic, characterized in that, include: S1. Load the malicious VPN IP address database and malicious VPN domain name database into the detection system memory, and perform data structuring preprocessing to convert the IP address and domain name information into the trie structure required by the DFA algorithm; S2. Configure uplink and downlink bidirectional traffic mirroring strategy to bypass network data packets to the detection system via port mirroring; S3. Read the mirror traffic data in real time through the packet capture interface, perform multi-layer protocol stack decoding, and extract application layer payload data; S4. Extract communication quintuple information, traffic content characteristics, and IP address location information from the parsed traffic; S5. Input the extracted destination IP address and domain name features into the pre-built DFA trie for fast matching: S5.

1. If the DFA match is successful, an alarm process is triggered, a security event object is generated, and complete traffic data within the time window before and after the malicious session is saved. S5.

2. If DFA matching fails, the JA3 and JA4 joint fingerprinting mechanism will be enabled to determine VPN traffic. S6. Automatically generate structured security response work orders and send alarm information to the security manager via email, SMS, and enterprise IM channels; S7. Track the handling process, start the SLA timer, and form a closed-loop process of detection, alarm, notification, and handling.

2. The method according to claim 1, characterized in that, The data structuring preprocessing includes converting the set of IP addresses and domain names into a Trie tree structure, and completing the construction of state nodes and the generation of jump tables.

3. The method according to claim 2, characterized in that, For IP addresses, a segmented Trie tree structure is used, which splits the IP address into four bytes in dotted decimal format for storage; for domain names, a character-level Trie tree structure is used, which stores each character in the domain name string as a tree node.

4. The method according to claim 1, characterized in that, The configured uplink and downlink bidirectional traffic mirroring strategy adopts the SPAN port mirroring strategy.

5. The method according to claim 1, characterized in that, The real-time reading of mirrored traffic data via packet capture interface uses PF_RING, DPDK, or libpcap packet capture interface for traffic collection.

6. The method according to claim 1, characterized in that, The traffic content characteristics include the SNI field in the TLS handshake message, the Host field, User-Agent field in the HTTP protocol, the certificate serial number, and certificate authority information; the IP address location information is obtained by querying the GeoIP geographic information database, including the country, province, city, and ISP operator information of the IP address.

7. The method according to claim 1, characterized in that, If the DFA match is successful, it is directly identified as known malicious VPN traffic, generating a packet capture file in pcap format or a structured JSON log, and solidifying the traffic communication link information, recording the source identifier, destination identifier, intermediate network nodes, session duration, packet sequence number, and byte stream statistics.

8. The method according to claim 1, characterized in that, The JA3 algorithm extracts the TLS version number, cipher suite list and its order, extended field list, elliptic curve type and dot format parameters from the TLS ClientHello message, and calculates and generates a hash fingerprint. The JA4 algorithm further incorporates HTTP / 2 frame structure features, QUIC protocol features, certificate chain information, and timestamp dimensions based on the JA3 algorithm to construct a highly distinguishable protocol fingerprint. By storing JA3 or JA4 fingerprints of known malicious VPNs in a pre-built VPN client fingerprint database, the Jaccard similarity between real-time traffic fingerprints and fingerprints in the database is calculated. If the similarity exceeds a preset threshold, it is determined to be VPN traffic.

9. The method according to claim 8, characterized in that, It also includes adding the domain name information and IP address corresponding to traffic that successfully matches JA3 or JA4 fingerprints to the malicious VPN domain name database and IP database, and reinitializing the DFA dictionary tree.

10. A multi-stage detection system for malicious VPN traffic, characterized in that, include: The system includes a feature library loading module, a traffic mirroring configuration module, a traffic capture and parsing module, a metadata extraction module, a DFA matching module, a JA3 / JA4 fingerprint detection module, an alarm and link solidification module, a work order and notification module, and a closed-loop management module for handling; among which: The feature library loading module is used to load the malicious VPN IP address library and the malicious VPN domain name library, and convert the IP address and domain name information into a trie structure required by the DFA algorithm; the feature library loading module is specifically used to convert the set of IP addresses and domain names into a Trie tree structure, and complete the construction of state nodes and the generation of jump tables; The traffic mirroring configuration module is used to configure uplink and downlink bidirectional traffic mirroring policies at network nodes, bypassing network data packets to the detection system; the traffic mirroring configuration module adopts the SPAN port mirroring policy; The traffic capture and parsing module is used to read mirrored traffic data in real time through the packet capture interface, perform multi-layer protocol stack decoding, and extract application layer payload; the traffic capture and parsing module uses PF_RING, DPDK or libpcap packet capture interface for traffic collection; The metadata extraction module is used to extract communication quintuples, traffic content features, and IP address location information. The traffic content features include the SNI field in the TLS handshake message, the Host field, User-Agent field, certificate serial number, and certificate authority information in the HTTP protocol. The IP address location information is obtained by querying the GeoIP geographic information database or the ASN routing database, and includes the country, province, city, and ISP operator information of the IP address. The DFA matching module is used to input the extracted traffic features into the DFA trie for fast matching; if the match is successful, the alarm process is triggered directly. The JA3 / JA4 fingerprint detection module is used to perform JA3 / JA4 fingerprint identification on traffic that fails to match the DFA to determine whether it is VPN traffic. The JA3 / JA4 fingerprint detection module also includes a pre-built VPN client / server fingerprint database, which is used to store the JA3 / JA4 fingerprints of known malicious VPNs and to determine whether the traffic is VPN traffic through Jaccard similarity calculation or threshold matching. The JA3 / JA4 fingerprint detection module is also used to add the domain name information and IP address corresponding to the traffic with successfully matched fingerprints to the malicious VPN domain name database and IP database, triggering the re-initialization of the DFA trie. The alarm and link solidification module is used to trigger alarms and save traffic communication link information when malicious traffic is detected, generate packet capture files in pcap format or structured JSON logs, and record source identifier, destination identifier, intermediate network nodes, session duration, data packet sequence number and byte stream statistics. The work order and notification module is used to generate safety handling work orders and send alarm information via email, SMS, and enterprise IM channels; the work order and notification module connects to third-party platforms through a Webhook interface to push card messages and supports one-click jump to the work order details page; The closed-loop management module is used to track the handling process, start the SLA timer and include an overdue escalation mechanism. When a work order is not processed within the preset time limit, it is automatically escalated to the next higher level of management for handling supervision.

Images