HTTPS traffic decryption method based on the technology of seamlessly differentiating original TCP connections

Abstract

This invention discloses an HTTPS traffic decryption method based on the technology of seamlessly differentiating original TCP connections. The method includes: relying on the user's internet access system, embedding the HTTPS domain name to be decrypted, and distributing backflow traffic to the DPI device; a forwarding module processes the traffic, and a monitoring module establishes two types of HTTPS connections; identifying the uplink traffic, matching the domain name, distinguishing whether decryption is required, and processing it accordingly (either by injection or differentiation) to complete the data exchange between the two types of connections. This method requires no proxy nodes or complex configurations; it can achieve HTTPS session hijacking and decryption simply by accessing normal internet traffic. It can be deployed anywhere, including universities, enterprises, provincial network backbones, and international nodes. Without interfering with established TCP connections, it seamlessly intervenes and differentiates connections during the TLS handshake phase, reusing the original TCP session five-tuple to establish an externally imperceptible bidirectional HTTPS channel, completing the decryption and forwarding of encrypted data.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and specifically to a method for decrypting HTTPS traffic based on the technology of seamlessly differentiating original TCP connections. Background Technology

[0002] While current man-in-the-middle interception decryption schemes can decrypt HTTPS traffic, they exhibit several significant drawbacks in practical applications. From a deployment perspective, this scheme requires precise deployment of proxy nodes at the exit point of the intercepted client. This means that the network architecture must configure a separate physical or virtual proxy entity for each client that needs monitoring. Taking an enterprise-level network environment as an example, if traffic monitoring is required for hundreds of terminal devices, not only must a proxy node be deployed at the exit gateway of each device, but the TCP communication configuration between the proxy node and the client must also be ensured to be accurate. This involves the client manually modifying system network settings, configuring the proxy server IP and port, and in some scenarios, installing and trusting the root certificate of the proxy node to avoid browser security warnings. Such operations have a high technical threshold for ordinary users, and large-scale deployments are prone to configuration inconsistencies, such as differences in proxy configuration interfaces across different operating systems (Windows / macOS / Linux) and priority conflicts between browser and system proxy settings. This makes the solution difficult to promote in network environments with varying skill levels, such as enterprises and schools, and it can only be applied to personally controlled devices or small-scale laboratory scenarios.

[0003] In terms of usability, the complex pre-configuration process is a major pain point. In traditional network access scenarios, users only need to configure DNS or gateway to access the network. However, this solution requires users to actively participate in the proxy node connection process: first, they need to manually specify the proxy server address through command line or graphical interface; second, they need to ensure that the proxy node's port is not blocked by the firewall; and finally, they need to handle certificate trust issues during HTTPS connections. For non-technical users, configuration errors in any step can lead to traffic takeover failure, and troubleshooting requires multiple checks at the network layer, transport layer, and application layer, significantly increasing maintenance costs. For example, in public Wi-Fi scenarios, requiring each access user to configure the proxy node themselves is obviously impractical, which directly prevents this solution from being implemented in open network environments.

[0004] From a network behavior perspective, the centralized forwarding mechanism of proxy nodes leads to highly homogenized traffic characteristics. When all client traffic is forwarded through proxy nodes, all requests received by the target server originate from the same single IP address of the proxy node. This abnormal traffic aggregation is easily identified by the target server's anti-proxy detection system. Taking e-commerce platforms or social media as examples, their risk control systems typically trigger security alerts for high-frequency access from the same IP address, causing the proxy node IP to be quickly blacklisted, thus affecting normal traffic forwarding functions. Furthermore, the uniformity of source IPs can also lead to the failure of network auditing. When it is necessary to trace the network behavior of a specific client, only the proxy node can be located, not the real client, which will result in the loss of critical information in network security incident investigations. A more far-reaching impact is that this proxy characteristic may trigger intervention from ISPs (Internet Service Providers), such as using DPI (Deep Packet Inspection) technology to identify the communication patterns of proxy nodes and implement traffic blocking, causing the entire decryption scheme to fail.

[0005] Further analysis of the inherent flaws in the technical architecture reveals that the proxy node, acting as an intermediate hub for traffic forwarding, essentially becomes a single point of bottleneck in network communication. As the number of clients increases or traffic load rises, the proxy node's CPU computing power (for SSL / TLS encryption / decryption), memory bandwidth (for data caching), and network I / O capabilities may become performance bottlenecks, leading to increased data forwarding latency or even packet loss. Simultaneously, the proxy node faces a dual security challenge: ensuring it is not compromised by attacks from clients or target servers, while simultaneously preventing the unauthorized theft of decrypted plaintext data within the node. In existing solutions, the proxy node is typically exposed on the communication link between the client and the ISP; its own security level directly determines the reliability of the entire decryption system. Complex network attacks (such as variants of man-in-the-middle attacks exploiting SSL protocol vulnerabilities) may bypass the proxy node's security mechanisms, causing decryption failure or data leakage. Summary of the Invention

[0006] The purpose of this invention is to provide a method for decrypting HTTPS traffic based on the technology of non-perceptible differentiation of original TCP connections, so as to solve the problems in the prior art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an HTTPS traffic decryption method based on non-perceptible differentiation of original TCP connections, the method comprising:

[0008] S1. Based on the existing user Internet access system, the HTTPS domain name information to be decrypted is pre-built into the user Internet access system, and the IP address of the corresponding domain name is obtained by listening to DNS response messages. The IP address is dynamically sent to the DPI device so that the DPI device can return the bidirectional traffic of the corresponding IP address to the user Internet access system.

[0009] S2. The forwarding module in the user's internet access system receives the traffic returned by the DPI device. For business traffic and / or TCP three-way handshake traffic that it does not care about, it returns them to the DPI device and forwards them along the original line in the user's internet access system.

[0010] S3. The monitoring module in the user's internet access system deploys a local HTTPS service, and establishes Client_HTTPS connection and ISP_HTTPS connection in the HTTPS service of the monitoring module;

[0011] S4. The forwarding module in the user's internet access system identifies each uplink HTTPS traffic, extracts the corresponding SNI information based on the ClientHello message in the first packet of the TLS handshake, and matches the SNI information with the pre-set list of domain names to be decrypted.

[0012] S5. Traffic from the domains not matched in S4 is determined to be traffic that does not need to be processed and is injected back to the DPI device as is, so that the DPI device can continue to forward it along the original line.

[0013] S6. Determine the traffic of the domain name list matched in S4 as the session traffic that needs to be decrypted, split the session into two connection differentiation processes, and connect the session differentiation that the client is unaware of to the Client_HTTPS connection to complete the data interaction between the Client_HTTPS connections.

[0014] S7. The monitoring module seamlessly integrates with the ISP_HTTPS connection as an HTTPS client, completing data interaction between the ISP_HTTPS connection and the ISP_HTTPS connection.

[0015] Furthermore, in S3, the local IP port for listening is 192.168.1.1:1234.

[0016] Furthermore, S3 includes:

[0017] S301. Record the bidirectional session data of the original quintuple to the session hash table;

[0018] S302. Temporarily cache the ClientHello message and record SN_User=ClientHello.SN-1; SN_ISP=ClientHello.AN-1;

[0019] S303. Establish a Client_TCP connection between the client and the monitoring module;

[0020] S304. Establish a Client_HTTPS connection between the monitoring module and the client.

[0021] Furthermore, S303 includes:

[0022] S3031. Assign intranet 5-tuple information to the session in S301. The destination IP port of all assigned intranet 5-tuple data is the HTTPS service IP port that the monitoring module listens on.

[0023] S3032. The forwarding module constructs a SYN packet, in which the SN value is set to SN_User; the five-tuple information is the internal network five-tuple information, and sends it to the monitoring module.

[0024] S3033, The monitoring module responds to the SYN packet by returning a SYN_ACK message. The forwarding module records the SN information of the SYN_ACK message as SN_Monitor_ISP, and finally calculates the SN offset of the Client_HTTPS connection as SN_Difference_Client = SN_Monitor_ISP - SN_ISP;

[0025] S3034. The forwarding module constructs an ACK message to respond to the SYN_ACK message.

[0026] Furthermore, S304 includes:

[0027] S3041. The forwarding module modifies the cached ClientHello packet 5-tuple to the internal network 5-tuple information, and at the same time modifies the packet AN information AN=AN+SN_Difference_Client; and sends the modified packet to the monitoring module.

[0028] S3042. The monitoring module responds normally to the TLS handshake request and sends a ServerHello message to the forwarding module.

[0029] S3043. After receiving the ServerHello message, the forwarding module modifies the internal network 5-tuple information back to the original 5-tuple information and modifies the SN information SN=SN-SN_Difference_Client; it then injects the modified message back to the DPI device and finally sends it to the client.

[0030] S3044. Subsequent business messages following the client's ServerHello response message are routed back to the forwarding module via DPI. The forwarding module matches the session hash with the five-tuple information to find the session data, modifies the original five-tuple information to the internal network five-tuple information, and modifies the message AN information AN=AN+SN_Difference_Client. The modified message is then sent to the monitoring module.

[0031] S3045. Repeat S3042-S3043 above to process other TLS handshake messages.

[0032] Furthermore, S3 also includes:

[0033] S305. Establish an ISP_TCP connection between the monitoring module and the target ISP;

[0034] S306. Establish an ISP_HTTPS connection between the target ISP and the monitoring module.

[0035] Furthermore, S305 includes:

[0036] S3051, the monitoring module acts as an HTTPS client, initiating an HTTPS connection request to the target ISP, where the target IP and port use the internal network five-tuple information;

[0037] S3052. The forwarding module receives the TCP connection SYN packet initiated by the monitoring module, records the SN information of the SYN packet as SN_Monitor_User, and finally calculates the SN offset of the ISP_HTTPS connection SN_Difference_ISP=SN_Monitor_User-SN_User;

[0038] S3053. The forwarding module constructs a SYN_ACK message for the SYN packet, with the SN value set to SN_ISP, and sends it to the monitoring module.

[0039] S3054, The monitoring module protocol stack responds with a SYN_ACK message and then replies with an ACK message.

[0040] Furthermore, S306 includes:

[0041] S3061. The monitoring module sends a ClientHello message to the forwarding module;

[0042] S3062. The forwarding module receives the HTTPS traffic sent to the ISP by the monitoring module, converts the five-tuple information into public network five-tuple information, and modifies the SN information SN=SN-SN_Difference_ISP, injects it back to the DPI device and finally sends it to the ISP.

[0043] When the S3063 ISP responds to HTTPS traffic and returns to the forwarding module via the DPI device, the forwarding module matches the session hash with the five-tuple information to find the session data, modifies the original five-tuple information to the internal network five-tuple information, and modifies the packet AN information AN=AN+SN_Difference_ISP; then sends the modified packet to the monitoring module.

[0044] S3064. Repeat S3061-S3063 above to complete the establishment of the ISP_HTTPS connection.

[0045] Furthermore, after seamlessly integrating into the HTTPS connection with the client and ISP via S6 and S7, the forwarding module converts the continuous five-tuple data of the return traffic into intranet five-tuple information and maps the SN / AN information to the SN / AN information of the intranet session. It then restores the original five-tuple information and SN / AN of the response data from the monitoring module and injects it back into the link, thus establishing a bidirectional HTTPS communication channel between the monitoring module and the client and ISP.

[0046] Furthermore, the monitoring module, acting as an HTTPS terminal that communicates simultaneously with both the client and the ISP, is responsible for sending and receiving decrypted traffic from bidirectional HTTPS sessions and then re-encrypting and forwarding the traffic bidirectionally, ensuring that the communication of the original data is not affected while the data is decrypted.

[0047] The aforementioned technical solution overcomes the bottlenecks of traditional methods that rely on proxy nodes and complex configurations. Leveraging a self-developed traffic identification engine, it directly enables data interaction based on normal internet traffic from user terminals. This technology requires no additional network equipment or complex modifications to underlying protocols; simply deploying the device on the network egress link ensures compatibility with both IPv4 and IPv6 dual-stack environments, achieving seamless coverage of university education networks, enterprise campus networks, provincial backbone networks, and even international internet egress points. Its plug-and-play nature significantly reduces deployment costs and maintenance complexity, ensuring stable 24 / 7 operation even in dynamically changing network topologies.

[0048] At the technical implementation level, this solution employs a strategy combining Deep Packet Inspection (DPI) and dynamic traffic analysis. After the TCP connection completes the three-way handshake, the intelligent judgment module monitors the ClientHello and ServerHello messages during the TLS handshake phase in real time. Once HTTPS traffic matching preset rules is detected, the system initiates a seamless intervention mechanism: through session renegotiation technology, without interrupting the original connection, it uses the TCP session 5-tuple (source IP, destination IP, source port, destination port, protocol number) information to create a shadow channel completely identical to the original connection. This channel adopts a zero-trust architecture design, presenting itself as normal encrypted traffic to the outside world, while simultaneously performing bidirectional proxying of the SSL / TLS protocol stack in the background, dynamically parsing symmetric and asymmetric keys, and ultimately restoring encrypted data to plaintext content.

[0049] The core advantage of this technology lies in its non-intrusive nature to network communication, ensuring a completely seamless user experience without page lag, certificate alerts, or other anomalies. Through traffic mirroring and load balancing, the system can handle millions of concurrent connections per node, achieving a decryption success rate of up to 99.99% while guaranteeing data integrity. Whether dealing with highly encrypted scenarios such as financial payments and online education, or handling VPN encrypted traffic from multinational corporations, this solution can accurately identify and quickly decrypt data, providing robust data support for applications such as network security auditing, data compliance testing, and threat intelligence analysis. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0051] Figure 1 A flowchart illustrating the process of establishing a SYN for a Client-HTTPS connection.

[0052] Figure 2 A flowchart illustrating the process of establishing a SYN_ACK connection for a Client-HTTPS connection.

[0053] Figure 3 A flowchart illustrating the process of establishing an ACK for a Client-HTTPS connection.

[0054] Figure 4 A flowchart illustrating the process of establishing a ClientHello for a Client-HTTPS connection.

[0055] Figure 5 A flowchart illustrating the process of establishing a ServerHello connection for a Client-HTTPS connection.

[0056] Figure 6 A flowchart illustrating the process of establishing a SYN for an ISP-HTTPS connection.

[0057] Figure 7 A flowchart illustrating the process of establishing a SYN_ACK connection for an ISP-HTTPS connection.

[0058] Figure 8 A flowchart illustrating the process of establishing an ACK for an ISP-HTTPS connection.

[0059] Figure 9 A flowchart illustrating the process of establishing a ClientHello for an ISP-HTTPS connection.

[0060] Figure 10 A flowchart illustrating the process of establishing a ServerHello connection for an ISP-HTTPS connection.

[0061] Figure 11 This is a flowchart illustrating the bidirectional data decryption and forwarding request process. Detailed Implementation

[0062] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0063] Among them, DPI devices are network element devices that are connected in series in the link to perform bidirectional data forwarding and also have the function of bidirectional traffic backflow and injection to a specified IP.

[0064] Backflow: The DPI device on the serial link forwards the specified traffic to this system and blocks further forwarding to the client or ISP, indicating that the specified traffic is backflowed.

[0065] Note: This system forwards traffic to the DPI device, expecting the DPI device to forward the traffic normally to the client or ISP.

[0066] SN (SequenceNumber): TCP sequence number, initially randomly generated by the local end.

[0067] AN (Acknowledgment number): Acknowledgment number, calculated based on the peer's SN number after receiving data from the peer.

[0068] like Figures 1-11 As shown, an HTTPS traffic decryption method based on the technology of seamlessly differentiating the original TCP connection includes:

[0069] S1. Relying on the existing user internet access system, the HTTPS domain name information to be decrypted is pre-built into the user internet access system. By listening to DNS response messages, the IP address of the corresponding domain name is obtained, and the IP address is dynamically distributed to the DPI device, so that the DPI device can return bidirectional traffic to the user internet access system for the corresponding IP address. Here, "unnoticeable" is differentiated into being unnoticeable to the client and being unnoticeable to the ISP.

[0070] Specifically, the user internet access system, as the foundational platform for the entire traffic decryption process, requires a core architecture with high compatibility and scalability. Before actual deployment, technicians pre-install the HTTPS domain information to be decrypted into the user internet access system's storage module in the form of a list or database records using the system configuration interface or dedicated management tools. These domains to be decrypted are typically selected based on specific security monitoring, data analysis, or compliance audit requirements, covering websites that may involve the transmission of sensitive information or potential security threats.

[0071] During system operation, the network interface module continuously listens for DNS (Domain Name System) response messages on the network. DNS, acting as the internet's "phone book," is responsible for converting human-readable domain names into computer-readable IP addresses. When a user enters an HTTPS domain name into their browser to initiate an access request, the local DNS server receives the request, performs domain name resolution, and ultimately encapsulates the resolved IP address in a DNS response message and returns it to the user's device. The user's internet access system captures these DNS response messages by listening on the port and uses message parsing technology to extract the IP address corresponding to the target domain name from the messages.

[0072] To ensure that the acquired IP addresses can be accurately used for traffic processing, the system performs validity verification and format checks on the extracted IP addresses, excluding erroneous or invalid IP address records. Verified IP addresses are then stored in the system's cache and dynamically distributed to DPI (Deep Packet Inspection) devices via specific communication protocols (such as SNMP, Simple Network Management Protocol; or a custom API interface).

[0073] As a key node in traffic processing, the DPI device possesses powerful traffic identification and forwarding capabilities. Upon receiving an IP address from a user's internet system, the DPI device creates or updates corresponding rule entries in its traffic processing rule base. These rule entries define the traffic processing strategy for a specific IP address, requiring the DPI device to, when detecting bidirectional traffic related to that IP address (including request traffic sent by the user to the server and response traffic returned by the server to the user), use traffic mirroring or redirection techniques to completely redirect this traffic back to the designated receiving port of the user's internet system. During this process, the DPI device accurately identifies and extracts the target traffic by performing real-time analysis and matching of network packet header information (such as source IP address, destination IP address, port number, etc.), ensuring that only traffic related to the target IP address is accurately transmitted back, avoiding interference from irrelevant traffic, thereby providing an accurate and complete raw data foundation for subsequent HTTPS traffic decryption.

[0074] S2. The forwarding module in the user's internet access system receives the traffic returned by the DPI device. For business traffic and / or TCP three-way handshake traffic that it is not interested in, it returns them to the DPI device and forwards them along the original line in the user's internet access system.

[0075] Specifically, the forwarding module only processes target HTTPS traffic, directly injecting non-target traffic back into the system, reducing CPU time spent parsing useless data. It eliminates the need to maintain session tables, SN offsets, and other state data for non-target traffic, reducing memory and bandwidth pressure. It prevents non-target traffic from "detouring" within the user's internet system, reducing bandwidth consumption on internal links and reserving channels for target HTTPS traffic. Injected traffic is forwarded "on the original line," maintaining the original IP route, MAC address, and transmission path. The forwarding module records the sequence number (SN) and acknowledgment number (AN) of SYN / SYN+ACK / ACK packets. After injection into the DPI device, subsequent packets can still correctly match the connection state, preventing "connection reset" RST errors on the client. Non-target traffic does not enter the HTTPS service processing chain of the monitoring module, preventing abnormal traffic from impacting the decryption core module and improving system robustness.

[0076] S3. The monitoring module in the user's internet access system deploys a local HTTPS service, and establishes Client_HTTPS connection and ISP_HTTPS connection in the HTTPS service of the monitoring module;

[0077] Furthermore, in S3, the local IP port for listening is 192.168.1.1:1234.

[0078] Specifically, listening on the local IP port supports setting access rules via iptables / firewalld to allow communication only within the intranet segment where the forwarding module is located.

[0079] Furthermore, S3 includes:

[0080] S301. Record the bidirectional session data of the original quintuple into the session hash table.

[0081] Specifically, the original 5-tuple includes the source IP, source port, destination IP, destination port, and protocol, where the protocol is a transport layer protocol, such as TCP. The hash table uses either FNV-1a or MurmurHash3 algorithms to reduce the probability of hash collisions.

[0082] Bidirectional session data synchronization records the session status in both directions: from the client to the ISP and from the ISP to the client, ensuring traceability of both traffic. Client-initiated sessions use the client's address as the source IP and the ISP's server as the destination IP; ISP-returned sessions use the ISP's server as the source IP and the client's address as the destination IP, matched by reversing the 5-tuple.

[0083] By using the original 5-tuple as the key, the ClientHello cached data is associated with the session table entry, ensuring that subsequent steps can quickly query information such as offset and intranet 5-tuple.

[0084] S302. Temporarily cache the ClientHello message and record SN_User=ClientHello.SN-1 and SN_ISP=ClientHello.AN-1;

[0085] Specifically, SN_User = ClientHello.SN - 1, where ClientHello.SN is the TCP sequence number sent by the client; the reason for subtracting 1 is that in the TCP protocol, a SYN packet occupies one sequence number, and subsequent data starts from SN+1. Therefore, SN_User represents "the next expected sequence number to be received". SN_ISP = ClientHello.AN - 1 is essentially the reverse application of the TCP protocol's "acknowledgment number = received SN+1"—by using the acknowledgment number carried by the client in ClientHello (acknowledgment of the server's SYN+ACK), the sequence number (its own ISN) when the server first sends data is deduced. This formula is the fundamental rule for generating server-side sequence numbers in HTTPS communication and is also the core technical basis for maintaining TCP state when the monitoring module intervenes.

[0086] SN_ISP = ClientHello.AN - 1, where ClientHello.AN is the acknowledgment number carried by the client; SN_ISP is used to compare with the sequence number of the SYN_ACK message responded by the monitoring module to calculate the SN offset.

[0087] Accurate calculation of SN_User and SN_ISP is fundamental to establishing a Client_TCP connection. Calculation errors can lead to the monitoring module's SYN_ACK response being interpreted as invalid by the client, triggering an RST reset, and subsequent SN / AN conversion errors in the TLS handshake message, ultimately causing HTTPS connection failure.

[0088] This step, through precise session state management and sequence number calculation, lays the data foundation for the seamless differentiation of subsequent Client_HTTPS and ISP_HTTPS connections, and is one of the core technical steps to achieve transparent decryption of HTTPS traffic.

[0089] Instance verification

[0090] Assumption:

[0091] Client ISN(c) = 1000, Server ISN(s) = 2000.

[0092] 1. The three-way handshake process:

[0093] The client sends a SYN: SN=1000, AN=0 (no data has been acknowledged);

[0094] The server responded with SYN+ACK: SN=2000, AN=1000+1=1001;

[0095] The client sends ACK: SN=1000+1=1001, AN=2000+1=2001.

[0096] 2. When ClientHello sends:

[0097] Client TCP packet: SN=1001 (first data after the three-way handshake), AN=2001 (i.e. Server_ISN+1=2000+1).

[0098] Therefore, ClientHello.AN=2001.

[0099] 3. When the server replies with ServerHello:

[0100] The server's TCP packet SN must be its own ISN=2000, that is: SN_ISP=2000=ClientHello.AN-1=2001-1, the formula is true.

[0101] S303. Establish a Client_TCP connection between the client and the monitoring module;

[0102] Furthermore, S303 includes:

[0103] S3031. Assign intranet 5-tuple information to the session in S301. The destination IP port of all assigned intranet 5-tuple data is the HTTPS service IP port that the monitoring module listens on.

[0104] S3032. The forwarding module constructs a SYN packet, in which the SN value is set to SN_User; the five-tuple information is the internal network five-tuple information, and sends it to the monitoring module.

[0105] S3033. The monitoring module responds to the SYN packet by returning a SYN_ACK message. The forwarding module records the SN information of the SYN_ACK message as SN_Monitor_ISP, and finally calculates the SN offset of the Client_HTTPS connection: SN_Difference_Client = SN_Monitor_ISP - SN_ISP. Here, SN_ISP is the ISP-side session sequence number (SequenceNumberofISP); SN_Monitor_ISP is the session sequence number between the monitoring module and the ISP (SequenceNumberofMonitor - ISPConnection).

[0106] S3034. The forwarding module constructs an ACK message to respond to the SYN_ACK message.

[0107] Specifically, through the mapping between the internal network 5-tuple allocation and the original 5-tuple, the client's traffic is transparently redirected to the monitoring module, but it always believes it is communicating with the target ISP server. SN offset calculation ensures seamless connection of the TCP sequence numbers between the client and the monitoring module, splitting the original connection between the client and the ISP into two independent connections: client to monitoring module (Client_TCP connection) and monitoring module to ISP (ISP_TCP connection). The monitoring module can decrypt the traffic and perform in-depth analysis without interrupting the original communication.

[0108] Meanwhile, the SN offset ensures that: the SN of the message sent by the client is compatible with the protocol stack of the monitoring module after conversion; and the SN of the message returned by the monitoring module, after restoration, meets the client's expectations. The internal network 5-tuple allocation adopts a port pool round-robin algorithm to avoid port conflicts, and the connection establishment delay is <1ms. Zero-copy SYN packet construction reduces memory copying and improves message sending efficiency.

[0109] S303 achieves seamless TCP connection deconstruction through "five-tuple remapping + precise sequence number control." Its core technology involves building a transparent decryption proxy path between the client and the monitoring module. This mechanism not only ensures efficient decryption of HTTPS traffic but also maintains the integrity and reliability of the original communication at the network and transport layers, making it a key element in achieving "seamless monitoring" within the entire technical solution.

[0110] S304. Establish a Client_HTTPS connection between the monitoring module and the client.

[0111] Furthermore, S304 includes:

[0112] S3041. The forwarding module modifies the cached ClientHello packet 5-tuple to the internal network 5-tuple information, and at the same time modifies the packet AN information AN=AN+SN_Difference_Client; and sends the modified packet to the monitoring module.

[0113] S3042. The monitoring module responds normally to the TLS handshake request and sends a ServerHello message to the forwarding module.

[0114] S3043. After receiving the ServerHello message, the forwarding module modifies the internal network 5-tuple information back to the original 5-tuple information and modifies the SN information SN=SN-SN_Difference_Client; it then injects the modified message back to the DPI device and finally sends it to the client.

[0115] S3044. Subsequent business messages following the client's ServerHello response message are routed back to the forwarding module via DPI. The forwarding module matches the session hash with the five-tuple information to find the session data, modifies the original five-tuple information to the internal network five-tuple information, and modifies the message AN information AN=AN+SN_Difference_Client. The modified message is then sent to the monitoring module.

[0116] S3045. Repeat S3042-S3043 above to process other TLS handshake messages.

[0117] Specifically, by modifying the ClientHello 5-tuple and the AN value (S3041), the client's TLS handshake request is transparently redirected to the monitoring module, but it always believes that it is communicating with the target website.

[0118] The monitoring module responds to ServerHello (S3042) with a pre-configured certificate. When the client verifies the certificate: if it is a self-signed certificate, the client needs to pre-trust the certificate; if it is an internal CA certificate, the client can verify it normally, achieving "man-in-the-middle" decryption without warning.

[0119] The forwarding module converts between the original 5-tuple (client↔ISP) and the internal network 5-tuple (client↔monitoring module) in real time to ensure that: traffic sent by the client can be correctly routed to the monitoring module; and traffic returned by the monitoring module can be correctly injected back to the client.

[0120] Correct the SN / AN value of the TLS handshake message using the SN offset:

[0121] From client to monitoring module: AN+=SN_Difference_Client;

[0122] Monitoring module to client: SN-=SN_Difference_Client; ensures consistency of sequence number space during TLS handshake to avoid TCP layer out-of-order or reset.

[0123] The forwarding module uses zero-copy technology to process TLS handshake messages, avoiding multiple memory copies: after receiving ClientHello from the DPI device, it directly modifies the 5-tuple and AN values ​​in shared memory and sends them to the monitoring module; the same applies when processing ServerHello, reducing CPU and memory overhead.

[0124] Transparent decryption and monitoring of HTTPS traffic are achieved through a combination of "five-tuple redirection + sequence number correction + certificate proxy." The core technology is to direct encrypted traffic to a monitoring module for in-depth analysis without impacting the client experience. This mechanism not only addresses the pain point of traditional firewalls' inability to detect encrypted traffic but also ensures the feasibility of large-scale deployment in enterprise and carrier-grade networks through refined protocol processing.

[0125] Furthermore, S3 also includes:

[0126] S305. Establish an ISP_TCP connection between the monitoring module and the target ISP;

[0127] Furthermore, S305 includes:

[0128] S3051, the monitoring module acts as an HTTPS client, initiating an HTTPS connection request to the target ISP, where the target IP and port use the internal network five-tuple information;

[0129] S3052. The forwarding module receives the TCP connection SYN packet initiated by the monitoring module, records the SN information of the SYN packet as SN_Monitor_User, and finally calculates the SN offset of the ISP_HTTPS connection SN_Difference_ISP=SN_Monitor_User-SN_User;

[0130] S3053. The forwarding module constructs a SYN_ACK message for the SYN packet, with the SN value set to SN_ISP, and sends it to the monitoring module.

[0131] S3054, The monitoring module protocol stack responds with a SYN_ACK message and then replies with an ACK message.

[0132] Specifically, by establishing an independent ISP_TCP connection, an isolated channel is formed with the Client_TCP connection: client traffic flows back to the user's internet system through the DPI device, and is directed to the monitoring module via the forwarding module; the monitoring module, as a client, actively connects to the ISP to simulate real user behavior.

[0133] The communication between the monitoring module and the ISP is completed in the intranet environment, and the monitoring module cannot be directly accessed by the public network, reducing the risk of being attacked; even if there are vulnerabilities on the ISP side, attackers cannot directly access the monitoring module, protecting the security of the system's core components.

[0134] By calculating SN_Difference_ISP, the sequence number space mapping between the monitoring module and the ISP is realized: the SYN packet sent by the monitoring module has SN=SN_Monitor_User; the forwarding module constructs SYN_ACK with SN=SN_ISP(ClientHello.AN-1); ensuring that the TCP protocol stacks at both ends have consistent expectations for the sequence number.

[0135] The forwarding module simulates ISP behavior to construct SYN_ACK, enabling the monitoring module's protocol stack to complete the three-way handshake normally: the monitoring module believes that a connection has been established with the ISP and sends ClientHello according to the standard procedure; actual communication is relayed through the forwarding module to maintain the synchronization of the two ends.

[0136] By constructing SYN_ACK using a pre-calculated SN value, the round-trip delay of the traditional three-way handshake is avoided: after the monitoring module sends SYN, it immediately receives SYN_ACK from the forwarding module without waiting for the ISP's response; the connection establishment time is shortened from the traditional 2RTT to 1RTT, improving the efficiency of the HTTPS handshake.

[0137] Reuse the five-tuple information of the Client_TCP connection to reduce memory usage: Each session only needs to store the SN_Difference_ISP additionally, without having to repeatedly record the basic five-tuple data; memory utilization is improved when supporting millions of concurrent connections.

[0138] By simultaneously controlling the connection between the client and the ISP, the monitoring module can decrypt the bidirectional traffic between the client and the ISP, including request and response. After decryption, secondary encryption and forwarding are performed; otherwise, communication will be interrupted. This enables end-to-end traffic content auditing and identifies the risk of bidirectional data leakage.

[0139] When the monitoring module detects malicious traffic, it can proactively disconnect the ISP_TCP connection while maintaining the normal Client_TCP connection: returning a 503 ServiceUnavailable response to the client to avoid user perception of abnormality; and recording the characteristics of the attack traffic to improve the accuracy of subsequent detection.

[0140] Transparent communication between the monitoring module and the ISP is achieved through a combination of "active connection by the monitoring module + serial number mapping + proxy response from the forwarding module." The core technology involves building a decryption proxy path within the user's internet access system, enabling the monitoring module to interact with the ISP as a client while maintaining a seamless connection. This mechanism is crucial for achieving bidirectional decryption of HTTPS traffic, ensuring that the system can perform in-depth analysis and security protection of encrypted traffic without impacting user experience.

[0141] S306. Establish an ISP_HTTPS connection between the target ISP and the monitoring module.

[0142] Furthermore, S306 includes:

[0143] S3061. The monitoring module sends a ClientHello message to the forwarding module;

[0144] S3062. The forwarding module receives the HTTPS traffic sent to the ISP by the monitoring module, converts the five-tuple information into public network five-tuple information, and modifies the SN information SN=SN-SN_Difference_ISP, injects it back to the DPI device and finally sends it to the ISP.

[0145] When the S3063 ISP responds to HTTPS traffic and returns to the forwarding module via the DPI device, the forwarding module matches the session hash with the five-tuple information to find the session data, modifies the original five-tuple information to the internal network five-tuple information, and modifies the packet AN information AN=AN+SN_Difference_ISP; then sends the modified packet to the monitoring module.

[0146] S3064. Repeat S3061-S3063 above to complete the establishment of the ISP_HTTPS connection.

[0147] Through the cyclical processing of S3061-S3063, the monitoring module establishes an independent HTTPS connection with the ISP, forming a two-way transparent channel of "client-monitoring module-ISP": the monitoring module acts as an intermediate node, establishing encrypted connections with both the client and the ISP, decrypting and forwarding all traffic; neither the client nor the ISP is aware of the existence of the intermediate node, and the communication process is consistent with a direct connection.

[0148] The monitoring module can simultaneously acquire: plaintext HTTPS requests sent by the client (decrypted via Client_HTTPS connection); plaintext HTTPS responses returned by the ISP (decrypted via ISP_HTTPS connection); achieving true end-to-end traffic content auditing and detecting encryption layer threats that traditional firewalls cannot recognize.

[0149] Correct the SN / AN value using SN_Difference_ISP: from the monitoring module to the ISP: SN = SN_Difference_ISP; from the ISP to the monitoring module: AN = SN_Difference_ISP; ensure that the sequence numbers of the TCP protocol stack are consistent at both ends to avoid out-of-order or reset issues.

[0150] Transparent two-way decryption of HTTPS traffic is achieved through "five-tuple redirection + sequence number correction + two-way encrypted channel." The core technology is to build a seamless decryption proxy path between the client and the ISP. This mechanism not only solves the problem of traditional security devices being unable to detect encrypted traffic, but also ensures the system's stability and scalability in high-concurrency and complex network environments through refined protocol processing.

[0151] S4. The forwarding module in the user's internet access system identifies each uplink HTTPS traffic, extracts the corresponding SNI information based on the ClientHello message in the first packet of the TLS handshake, and matches the SNI information with the pre-set list of domain names to be decrypted.

[0152] Specifically, by extracting the SNI (ServerNameIndication) information from the ClientHello message, the system can: identify the specific domain name accessed by the user, rather than relying solely on the IP address; distinguish different domain names under the same IP; and overcome the limitations of traditional firewalls that only identify traffic based on IP / port. Even if the IP address corresponding to the domain name changes dynamically, it can still be accurately identified through SNI. Through SNI information extraction and matching, intelligent classification and differentiated processing of HTTPS traffic are achieved. The essence of this technology is finding the optimal balance between security protection and system performance. This mechanism not only improves the network security equipment's ability to detect encrypted traffic but also ensures that the user experience is not affected through fine-grained control, making it a key technical means for enterprise-level and carrier-level network security deployments.

[0153] S5. Traffic from the domains not matched in S4 is determined to be traffic that does not need to be processed and is injected back to the DPI device as is, so that the DPI device can continue to forward it along the original line.

[0154] Specifically, traffic to unmatched domains is directly injected back, avoiding invalid processing of non-target traffic by the system: there is no need to parse TLS handshakes, calculate SN offsets, or maintain session states; CPU utilization is reduced by more than 60%, memory usage is reduced by 50%, and resources are freed up for core decryption tasks.

[0155] By directly injecting unmatched traffic, intelligent scheduling of system resources and transparent maintenance of network communication are achieved. Essentially, this approach establishes a balance between security monitoring needs and network performance assurance. This design not only ensures the efficient operation of the HTTPS decryption system but also meets compliance and privacy requirements in the modern network environment by minimizing the scope of data processing, making it an indispensable part of enterprise-level security solutions.

[0156] S6. Determine the traffic of the domain name list matched in S4 as the session traffic that needs to be decrypted, split the session into two connection differentiation processes, and connect the session differentiation that the client is unaware of to the Client_HTTPS connection to complete the data interaction between the Client_HTTPS connections.

[0157] S7. The monitoring module seamlessly integrates with the ISP_HTTPS connection as an HTTPS client, completing data interaction between the ISP_HTTPS connection and the ISP_HTTPS connection.

[0158] Specifically, through S6 connection splitting, the original HTTPS session between the client and the ISP is transparently split into two independent connections: client to monitoring module (Client_HTTPS); monitoring module to ISP (ISP_HTTPS). The client does not need to install any software, and the communication process is completely consistent with a direct connection to the ISP. The monitoring module ensures the synchronization of the Client_HTTPS and ISP_HTTPS connection states by precisely controlling the TCP sequence number and TLS handshake parameters; it also maintains the continuity of the session key, preventing the decryption process from affecting the security of encrypted communication.

[0159] S6 and S7 achieve deep inspection and security protection of HTTPS traffic without affecting user experience through "connection differentiation" and "unobtrusive man-in-the-middle decryption". This combination of technologies solves the problem that traditional network security devices cannot effectively monitor encrypted traffic, and at the same time, through refined protocol processing and performance optimization, it ensures the feasibility and reliability of the system in high-concurrency and complex network environments.

[0160] Furthermore, after seamlessly integrating into the HTTPS connection with the client and ISP via S6 and S7, the forwarding module converts the continuous five-tuple data of the return traffic into intranet five-tuple information and maps the SN / AN information to the SN / AN information of the intranet session. It then restores the original five-tuple information and SN / AN of the response data from the monitoring module and injects it back into the link, thus establishing a bidirectional HTTPS communication channel between the monitoring module and the client and ISP.

[0161] Specifically, through dynamic mapping of the five-tuple with SN / AN information, the system achieves transparent proxying and bidirectional communication for TCP / TLS sessions. Its technical essence lies in building a seamless decryption proxy path between the client and the ISP. This mechanism not only solves the challenge of deep inspection of HTTPS traffic but also ensures the system's stability and scalability in high-concurrency, complex network environments through refined protocol processing. It is a core technology for enterprise-level and carrier-grade network security protection.

[0162] Furthermore, the monitoring module, acting as an HTTPS terminal that communicates simultaneously with both the client and the ISP, is responsible for sending and receiving decrypted traffic from bidirectional HTTPS sessions and then re-encrypting and forwarding the traffic bidirectionally, ensuring that the communication of the original data is not affected while the data is decrypted.

[0163] Specifically, the monitoring module's bidirectional encrypted forwarding mechanism achieves a balance between security detection and privacy protection. Its technical essence is to "see through" HTTPS traffic without affecting the original communication. This mechanism not only solves the pain point of traditional security devices being unable to detect encrypted traffic, but also ensures the feasibility of large-scale deployment of the system in enterprise and carrier-grade networks through refined protocol processing and performance optimization.

[0164] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for decrypting HTTPS traffic based on the technology of seamlessly differentiating original TCP connections, characterized in that: The method includes: S1. Based on the existing user Internet access system, the HTTPS domain name information to be decrypted is pre-built into the user Internet access system, and the IP address of the corresponding domain name is obtained by listening to DNS response messages. The IP address is dynamically sent to the DPI device so that the DPI device can return the bidirectional traffic of the corresponding IP address to the user Internet access system. S2. The forwarding module in the user's internet access system receives the traffic returned by the DPI device. For business traffic and / or TCP three-way handshake traffic that it does not care about, it returns them to the DPI device and forwards them along the original line in the user's internet access system. S3. The monitoring module in the user's internet access system deploys a local HTTPS service, and establishes Client_HTTPS connection and ISP_HTTPS connection in the HTTPS service of the monitoring module; S4. The forwarding module in the user's internet access system identifies each uplink HTTPS traffic, extracts the corresponding SNI information based on the ClientHello message in the first packet of the TLS handshake, and matches the SNI information with the pre-set list of domain names to be decrypted. S5. Traffic from the domains not matched in S4 is determined to be traffic that does not need to be processed and is injected back to the DPI device as is, so that the DPI device can continue to forward it along the original line. S6. Determine the traffic of the domain name list matched in S4 as the session traffic that needs to be decrypted, split the session into two connection differentiation processes, and connect the session differentiation that the client is unaware of to the Client_HTTPS connection to complete the data interaction between the Client_HTTPS connections. S7. The monitoring module seamlessly integrates with the ISP_HTTPS connection as an HTTPS client, completing data interaction between the ISP_HTTPS connection and the ISP_HTTPS connection.

2. The HTTPS traffic decryption method based on the non-perceptible differentiation of original TCP connections according to claim 1, characterized in that, In S3, the local IP port for listening is 192.168.1.1:1234.

3. The HTTPS traffic decryption method based on the non-perceptible differentiation of original TCP connections according to claim 1, characterized in that, S3 include: S301. Record the bidirectional session data of the original quintuple to the session hash table; S302. Temporarily cache the ClientHello message and record SN_User=ClientHello.SN-1; SN_ISP=ClientHello.AN-1; S303. Establish a Client_TCP connection between the client and the monitoring module; S304. Establish a Client_HTTPS connection between the monitoring module and the client.

4. The HTTPS traffic decryption method based on the non-perceptible differentiation of original TCP connections according to claim 3, characterized in that, S303 includes: S3031. Assign intranet 5-tuple information to the session in S301. The destination IP port of all assigned intranet 5-tuple data is the HTTPS service IP port that the monitoring module listens on. S3032. The forwarding module constructs a SYN packet, in which the SN value is set to SN_User; the five-tuple information is the internal network five-tuple information, and sends it to the monitoring module. S3033, The monitoring module responds to the SYN packet by returning a SYN_ACK message. The forwarding module records the SN information of the SYN_ACK message as SN_Monitor_ISP, and finally calculates the SN offset of the Client_HTTPS connection as SN_Difference_Client = SN_Monitor_ISP - SN_ISP; S3034. The forwarding module constructs an ACK message to respond to the SYN_ACK message.

5. The HTTPS traffic decryption method based on the non-perceptible differentiation of original TCP connections according to claim 3, characterized in that, S304 includes: S3041. The forwarding module modifies the cached ClientHello packet 5-tuple to the internal network 5-tuple information, and at the same time modifies the packet AN information AN=AN+SN_Difference_Client; and sends the modified packet to the monitoring module. S3042. The monitoring module responds normally to the TLS handshake request and sends a ServerHello message to the forwarding module. S3043. After receiving the ServerHello message, the forwarding module modifies the internal network 5-tuple information back to the original 5-tuple information and modifies the SN information SN=SN-SN_Difference_Client; it then injects the modified message back to the DPI device and finally sends it to the client. S3044. Subsequent business messages following the client's ServerHello response message are routed back to the forwarding module via DPI. The forwarding module matches the session hash with the five-tuple information to find the session data, modifies the original five-tuple information to the internal network five-tuple information, and modifies the message AN information AN=AN+SN_Difference_Client. The modified message is then sent to the monitoring module. S3045. Repeat S3042-S3043 above to process other TLS handshake messages.

6. The HTTPS traffic decryption method based on the non-perceptible differentiation of original TCP connections according to claim 1, characterized in that, S3 also includes: S305. Establish an ISP_TCP connection between the monitoring module and the target ISP; S306. Establish an ISP_HTTPS connection between the target ISP and the monitoring module.

7. The HTTPS traffic decryption method based on the non-perceptible differentiation of original TCP connections as described in claim 6, characterized in that, S305 includes: S3051, the monitoring module acts as an HTTPS client, initiating an HTTPS connection request to the target ISP, where the target IP and port use the internal network five-tuple information; S3052. The forwarding module receives the TCP connection SYN packet initiated by the monitoring module, records the SN information of the SYN packet as SN_Monitor_User, and finally calculates the SN offset of the ISP_HTTPS connection SN_Difference_ISP=SN_Monitor_User-SN_User; S3053. The forwarding module constructs a SYN_ACK message for the SYN packet, with the SN value set to SN_ISP, and sends it to the monitoring module. S3054, The monitoring module protocol stack responds with a SYN_ACK message and then replies with an ACK message.

8. The HTTPS traffic decryption method based on the non-perceptible differentiation of original TCP connections according to claim 1, characterized in that, S306 includes: S3061. The monitoring module sends a ClientHello message to the forwarding module; S3062. The forwarding module receives the HTTPS traffic sent to the ISP by the monitoring module, converts the five-tuple information into public network five-tuple information, and modifies the SN information SN=SN-SN_Difference_ISP, injects it back to the DPI device and finally sends it to the ISP. When the S3063 ISP responds to HTTPS traffic and returns to the forwarding module via the DPI device, the forwarding module matches the session hash with the five-tuple information to find the session data, modifies the original five-tuple information to the internal network five-tuple information, and modifies the packet AN information AN=AN+SN_Difference_ISP; then sends the modified packet to the monitoring module. S3064. Repeat S3061-S3063 above to complete the establishment of the ISP_HTTPS connection.

9. The HTTPS traffic decryption method based on the non-perceptible differentiation of original TCP connections according to claim 1, characterized in that, After seamlessly integrating into the HTTPS connection with the client and ISP via S6 and S7, the forwarding module converts the continuous five-tuple data of the return traffic into internal network five-tuple information and maps the SN / AN information to the SN / AN information of the internal network session. It then restores the original five-tuple information and SN / AN of the monitoring module's response data and injects it back into the link, thus establishing a bidirectional HTTPS communication channel between the monitoring module and the client and ISP.

10. The HTTPS traffic decryption method based on the non-perceptible differentiation of original TCP connections according to claim 1, characterized in that, The monitoring module, acting as an HTTPS terminal that communicates simultaneously with both the client and the ISP, is responsible for sending and receiving decrypted traffic from bidirectional HTTPS sessions and then re-encrypting and forwarding the traffic bidirectionally, ensuring that the communication of the original data is not affected while the data is decrypted.

Images