Hyper text transport protocol (HTTP) client system based on stage tracking and request processing method

Abstract

The invention provides an HTTP (Hyper Text Transport Protocol) client system based on stage tracking and a request processing method, and the system comprises a client engine which is used for coordinating the full life cycle execution of an HTTP request; the request builder is coupled with the client engine and is used for generating an HTTP (Hyper Text Transport Protocol) request containing a uniform resource identifier and a request method; the stage processor is coupled with the client engine and is used for executing domain name resolution, transmission control protocol connection establishment, transport layer security protocol handshake and HTTP protocol exchange in sequence; the tracker controller is respectively coupled with the client engine, the request builder and the stage processor, and is used for injecting and calling tracking logic in at least one key stage of the full life cycle of the HTTP request; through defining standardized and pluggable observation points of a request life cycle, the time consumption and the state of each sub-stage are accurately measured.

Description

An HTTP client system and request processing method based on phase tracing Technical Field

[0001] This invention relates to the field of network communication technology, and in particular to an HTTP client system and request processing method based on stage tracing. Background Technology

[0002] Traditional HTTP clients (such as cURL and reqwest) are "black boxes" or "grey boxes." When a network request fails, it is difficult to quickly locate the problem as to whether it is in the DNS, TCP connection, TLS handshake, or application layer. Furthermore, existing clients are tightly coupled internally, making it difficult to replace core components (such as using a custom DNS, a special proxy protocol, or a different TLS library). Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide an HTTP client system and request processing method based on stage tracing. By defining standardized and pluggable observation points for the request lifecycle, it accurately measures the time and status of each sub-stage (DNS resolution, TCP connection, TLS handshake, HTTP exchange), providing a standardized data framework for network quality monitoring, root cause location of faults, and performance bottleneck analysis.

[0004] In a first aspect, the present invention provides an HTTP client system based on stage tracing, comprising: a client engine for coordinating the entire lifecycle execution of an HTTP request; a request builder coupled to the client engine for generating an HTTP request containing a Uniform Resource Identifier (URI) and a request method; a stage processor coupled to the client engine for sequentially executing domain name resolution, Transmission Control Protocol (TCP) connection establishment, Transport Layer Security (TLS) handshake, and HTTP protocol exchange; and a tracer controller coupled to the client engine, the request builder, and the stage processor, for injecting and invoking tracing logic at at least one key stage in the entire lifecycle of the HTTP request; wherein the stage processor includes a pluggable domain name resolver component, a pluggable TCP connector component, and a pluggable TLS handshake component, and the tracer controller invokes tracing logic corresponding to each component to record the status and performance data of each stage.

[0005] Secondly, the present invention provides an HTTP client request processing method based on stage tracing, applied to the system described in the first aspect. The method includes: Step 1, generating an HTTP request through the request builder and associating it with a tracker; Step 2, calling the tracker's request start callback through the tracker controller; Step 3, sequentially executing the following stages through the stage processor, with the tracker controller calling the corresponding tracing callback at the beginning and end of each stage: a) Domain name resolution stage, resolving the hostname in the HTTP request through the pluggable domain name resolver component; b) Transmission Control Protocol (TCP) connection stage, establishing a connection with the resolved or preset network address through the pluggable TCP connector component; c) Transport Layer Security (TLS) handshake stage, establishing a secure channel on the connection through the pluggable TLS handshake component when the HTTP request uses a security protocol; d) HTTP exchange stage, sending the HTTP request and receiving a response through the secure channel or the connection; Step 4, summarizing the performance data of each stage recorded by the tracker to generate a request execution report.

[0006] The present invention provides one or more technical solutions, which have at least the following technical effects or advantages: By defining standardized, pluggable observation points for the request lifecycle, the present invention can precisely measure the time and status of each sub-stage (DNS resolution, TCP connection, TLS handshake, HTTP exchange) like a scalpel. This provides a standardized data framework for network quality monitoring, root cause location of faults, and performance bottleneck analysis, transforming network debugging from "guessing" to "data-driven".

[0007] This invention decouples the HTTP client into pluggable, independent components through three core traits: Resolver, TcpConnector, and TlsHandshaker. Users or enterprises can easily integrate their self-developed DNS services; adapt to special network environments or proxy protocols; replace TLS implementations to meet compliance requirements; and achieve separation of "policy" and "execution," resulting in highly flexible and maintainable architecture.

[0008] This invention initiates connections to multiple target IPs simultaneously, employing a "horse race mechanism" to select the first successful connection, greatly improving connection speed and success rate; failed connections are recorded independently, and other attempts are automatically canceled after a successful connection is established, saving system resources; timeouts are set separately for DNS, TCP, TLS, and global requests to avoid a single link freezing and affecting the whole, thus improving the overall resilience and reliability of the system.

[0009] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0010] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0011] Figure 1 is a flowchart of the method in Embodiment 2 of the present invention. Detailed Implementation

[0012] The overall concept of the technical solution in this application embodiment is as follows: I. Overall Architecture Design 1. Modular Layered Architecture Recorder layer: request tracing and statistics; Resolver layer: DNS resolution (supports custom DNS servers); TcpConnector layer: TCP connection establishment (supports parallel connections and proxies); TlsHandshaker layer: TLS handshake; Client layer: HTTP request sending and response processing.

[0013] 2. Core Features: Fully Asynchronous: Based on the tokio runtime; Pluggable Components: Each layer can be customized; Detailed Request Tracking: Accurately records the time consumption of each stage; Connection Strategy: Supports parallel connections and fallback mechanisms; Proxy Support: Through MuxProxy; HTTPS Support: Based on rustls; Automatic Header Setting: Automatically adds Host and User-Agent.

[0014] II. Core Component Details 1. Recorder Tracing System Rustpub trait Recorder: Send + Sync { / / Defines complete request lifecycle hooks fn on_start(&self, _request: &Request) {} fn on_dns_start(&self, _request: &Request, _resolver_config: &ResolverConfig, _host: &str) {} fn on_dns_done(&self, ...) {} / / ... more hooks} Function: Provides callback interfaces for each stage of the request StatsRecorder Implementation: Records the start / end time of each stage Collects error information Supports multi-stage parallel recording (such as multiple TCP connection attempts) 2. DNS Resolver Rust#[async_trait::async_trait]pub trait Resolver: Debug + Send + Sync {async fn resolve(&self, request: &Request) -> crate::Result <Vec <socketaddr>>;fn config(&self) -> &ResolverConfig;}DefaultResolver features: Based on trust-dns-resolver, supports custom DNS servers, supports IP address selection strategies (IPv4 / IPv6 priority) 3. TCP connector Rust#[async_trait::async_trait]pub trait TcpConnector: Debug + Send + Sync {async fn tcp_connect(&self, request: &Request, targets: Option <Vec <targetaddr>>) -> crate::Result <connection>DefaultTcpConnector Features: Parallel Connections: When there are multiple target addresses, connections are attempted in parallel. Fallback Mechanism: If the main connection fails, the following will happen after 3 seconds: I. Overall Architecture Design 1. Modular Layered Architecture Recorder Layer: Request tracing and statistics; Resolver Layer: DNS resolution (supports custom DNS servers); TcpConnector Layer: TCP connection establishment (supports parallel connections and proxy); TlsHandshaker Layer: TLS handshake; Client Layer: HTTP request sending and response processing.

[0015] 2. Core Features: Fully Asynchronous: Based on the tokio runtime; Pluggable Components: Each layer can be customized; Detailed Request Tracking: Accurately records the time consumption of each stage; Connection Strategy: Supports parallel connections and fallback mechanisms; Proxy Support: Through MuxProxy; HTTPS Support: Based on rustls; Automatic Header Setting: Automatically adds Host and User-Agent.

[0016] II. Core Component Details 1. Recorder Tracing System Rustpub trait Recorder: Send + Sync { / / Defines complete request lifecycle hooks fn on_start(&self, _request: &Request) {} fn on_dns_start(&self, _request: &Request, _resolver_config: &ResolverConfig, _host: &str) {} fn on_dns_done(&self, ...) {} / / ... more hooks} Function: Provides callback interfaces for each stage of the request StatsRecorder Implementation: Records the start / end time of each stage Collects error information Supports multi-stage parallel recording (such as multiple TCP connection attempts) 2. DNS Resolver Rust#[async_trait::async_trait]pub trait Resolver: Debug + Send + Sync {async fn resolve(&self, request: &Request) -> crate::Result <Vec <socketaddr>>;fn config(&self) -> &ResolverConfig;}DefaultResolver features: Based on trust-dns-resolver, supports custom DNS servers, supports IP address selection strategies (IPv4 / IPv6 priority) 3. TCP connector Rust#[async_trait::async_trait]pub trait TcpConnector: Debug + Send + Sync {async fn tcp_connect(&self, request: &Request, targets: Option <Vec <targetaddr>>) -> crate::Result <connection>DefaultTcpConnector features: Parallel connections: When there are multiple target addresses, connections are attempted in parallel. Fallback mechanism: If the main connection fails, the next connection is attempted after 3 seconds. Proxy support: Local binding via MuxProxy: Local IP address can be specified. 4. TLS handshake Rust#[async_trait::async_trait]pub trait TlsHandshaker: Debug + Send + Sync {async fn tls_handshake(&self, request: &Request, stream:Connection) -> crate::Result <TlsStream <connection>DefaultTlsHandshaker Features: Based on rustls, supports certificate verification skipping (for testing), supports ALPN protocol negotiation (HTTP / 1.1 and HTTP / 2), uses system certificate storage. III. Request Execution Flow 1. Complete Request Sequence text main() ↓ client.execute(request) ↓ ClientRef::execute() │├─▶ 1. recorder.on_start() # Request begins │├─▶ 2. DNS Resolution Phase │ ├─▶ recorder.on_dns_start() │ ├─▶ resolver.resolve() # Resolve domain name │ └─▶ recorder.on_dns_done() │├─▶ 3. TCP Connection Phase │ ├─▶ recorder.on_tcp_start() # Each target address │ ├─▶ tcp_connector.tcp_connect() # Establish connection │ └─▶ recorder.on_tcp_done() │├─▶ 4. TLS Handshake Phase (HTTPS Only) │ ├─▶ recorder.on_tls_start() │ ├─▶ tls_handshaker.tls_handshake() │ └─▶ recorder.on_tls_done() │ ├─▶ 5. HTTP Request Sending Phase │ ├─▶ recorder.on_send_request_start() │ ├─▶ Automatically set request headers │ ├─▶ Select HTTP version based on ALPN │ ├─▶ Send HTTP request │ └─▶ Receive response │ └─▶ 6. 2. Timeout Control: Each stage has independent timeout settings: dns_timeout: DNS resolution timeout; tcp_timeout: TCP connection timeout; tls_timeout: TLS handshake timeout. Overall request timeout: via request.timeout(). 3. Connection Strategy: Parallel Connections: When there are multiple IP addresses, connections are attempted in parallel. Fast Failure: As soon as one connection succeeds, a cancellation mechanism is used: other connection attempts are canceled using tokio's broadcast channel. IV. Code Example Analysis: 1. Usage Example: Rust #[tokio::main]pub async fn main() { let recorder = StatsRecorder::new(); / / Create a statistics recorder let client = ClientBuilder::new().build().unwrap(); / / Create client / / Send request and record statistics let _ = client.get("https: / / www.baidu.com").recorder(Box::new(recorder.clone())) / / Bind recorder.send().await.unwrap().text().await.unwrap();println!("{}", recorder.finish()) / / Output statistics}2. Configuration options RustClientBuilder::new().dns_timeout(Duration::from_secs(5)) / / DNS timeout.tcp_timeout(Duration::from_secs(10)) / / TCP connection timeout.tls_timeout(Duration::from_secs(5)) / / TLS handshake timeout.disable_auto_set_header(false) / / Whether to automatically set headers.local_dns(true) / / Whether to use local DNS.build(); Advantages of the above method: 1. Scalability: All core components are defined through traits; users can customize the implementation of any layer; supports multiple network environments and protocols.

[0017] 2. Detailed monitoring capabilities: Millisecond-accurate stage time recording; error tracking and logging; support for independent statistics for multiple concurrent connections.

[0018] 3. Intelligent connection management: accelerated parallel connections; efficient resource utilization (timely cancellation of unnecessary connections).

[0019] 4. Security: Uses rustls for TLS encryption; supports certificate verification; secure default configuration. Example

[0020] This embodiment provides an HTTP client system based on stage tracing, comprising: a client engine for coordinating the entire lifecycle execution of an HTTP request; a request builder coupled to the client engine for generating an HTTP request containing a Uniform Resource Identifier (URI) and a request method; a stage processor coupled to the client engine for sequentially executing domain name resolution, Transmission Control Protocol (TCP) connection establishment, Transport Layer Security (TLS) handshake, and HTTP protocol exchange; and a tracer controller coupled to the client engine, the request builder, and the stage processor for injecting and invoking tracing logic at at least one key stage in the entire lifecycle of the HTTP request. The stage processor includes a pluggable domain name resolver component, a pluggable TCP connector component, and a pluggable TLS handshake component. The tracer controller invokes tracing logic corresponding to each component to record the status and performance data of each stage.

[0021] In this embodiment, preferably, the tracker controller calls a tracker object that implements a unified tracking interface. The unified tracking interface defines callback methods that are called at specific nodes in the request lifecycle, including request start, domain name resolution start and completion, Transmission Control Protocol connection start and completion, Transport Layer Security (TLS) handshake start and completion, and HTTP request sending start.

[0022] In this embodiment, preferably, the tracker object is configured to record the start timestamp, completion timestamp, and error information of at least one key stage, and is capable of calculating stage duration and total request duration.

[0023] In this embodiment, preferably, the pluggable transmission control protocol connector component is configured to: when multiple target network addresses are received, adopt a parallel connection strategy, simultaneously initiate multiple transmission control protocol connection attempts at preset time intervals, and select the first successfully established connection for subsequent communication.

[0024] In this embodiment, preferably, the pluggable Transmission Control Protocol connector component supports connection through a proxy server, and the system further includes a timeout control module that sets independent timeout thresholds for the domain name resolution, the Transmission Control Protocol connection establishment, the Transport Layer Security (TLS) handshake, and the overall request execution.

[0025] In this embodiment, preferably, the pluggable transport layer security protocol handshake component is configured to negotiate the application layer protocol negotiation protocol during the handshake process, and automatically select to use Hypertext Transfer Protocol version 1.1 or Hypertext Transfer Protocol version 2 to communicate with the server based on the negotiation result.

[0026] In this embodiment, preferably, the client engine further includes an automatic request header management module, which is used to automatically detect and supplement necessary HTTP request header fields before sending a request.

[0027] Based on the same inventive concept, this application also provides a method corresponding to the system in Embodiment 1, as detailed in Embodiment 2.

[0028] Example 2, as shown in Figure 1, provides an HTTP client request processing method based on stage tracing, applied to the system described in Example 1. The method includes: Step 1: Generating an HTTP request using the request builder and associating it with a tracker; Step 2: Calling the tracker's request start callback through the tracker controller; Step 3: Sequentially executing the following stages through the stage processor, with the tracker controller calling the corresponding tracing callback at the beginning and end of each stage: a) Domain name resolution stage: resolving the hostname in the HTTP request using the pluggable domain name resolver component; b) Transmission Control Protocol (TCP) connection stage: establishing a connection with the resolved or preset network address using the pluggable TCP connector component; c) Transport Layer Security (TLS) handshake stage: establishing a secure channel on the connection when the HTTP request uses a TLS security protocol; d) HTTP exchange stage: sending the HTTP request and receiving a response through the secure channel or the connection; Step 4: Summarizing the performance data of each stage recorded by the tracker to generate a request execution report.

[0029] In this embodiment, preferably, during the transmission control protocol connection phase, when there are multiple alternative network addresses, a parallel competition connection mechanism is adopted. The pluggable transmission control protocol connector component initiates multiple connection attempts in parallel, and cancels the remaining pending connection attempts after the first connection is successfully established.

[0030] Since the method described in Embodiment 2 of this invention is a method used to implement the system of Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this method based on the system described in Embodiment 1 of this invention, and therefore will not be repeated here. All methods used in the system of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0031] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0032] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0033] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0034] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0035] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.< / connection> < / connection> < / targetaddr> < / socketaddr> < / connection> < / targetaddr> < / socketaddr>

Claims

1. An HTTP client system based on phase tracing, characterized in that, include: The client-side engine coordinates the entire lifecycle of HTTP requests. A request builder, coupled to the client engine, is used to generate HTTP requests containing a Uniform Resource Identifier and a request method; The stage processor, coupled to the client engine, is used to sequentially execute domain name resolution, Transmission Control Protocol connection establishment, Transport Layer Security Protocol handshake, and HTTP protocol exchange. The tracker controller, coupled to the client engine, the request builder, and the stage processor, is used to inject and invoke tracking logic at at least one key stage in the entire lifecycle of the HTTP request. The stage processor includes a pluggable domain name resolver component, a pluggable transport control protocol connector component, and a pluggable transport layer security protocol handshake component. The tracker controller invokes the tracking logic corresponding to each component to record the status and performance data of each stage.

2. The HTTP client system based on phase tracing according to claim 1, characterized in that, The tracker controller invokes a tracker object that implements a unified tracking interface, which defines callback methods that are invoked at specific nodes in the request lifecycle, including request start, domain name resolution start and completion, Transmission Control Protocol connection start and completion, Transport Layer Security (TLS) handshake start and completion, and HTTP request sending start.

3. The HTTP client system based on phase tracing according to claim 2, characterized in that, The tracker object is configured to record the start timestamp, completion timestamp, and error information of at least one key stage, and is capable of calculating stage duration and total request duration.

4. The HTTP client system based on phase tracing according to claim 1, characterized in that, The pluggable transmission control protocol connector component is configured to: when multiple target network addresses are received, adopt a parallel connection strategy, simultaneously initiate multiple transmission control protocol connection attempts at preset time intervals, and select the first successfully established connection for subsequent communication.

5. The HTTP client system based on phase tracing according to claim 1, characterized in that, The pluggable Transmission Control Protocol connector component supports connection via a proxy server. The system also includes a timeout control module that sets independent timeout thresholds for domain name resolution, Transmission Control Protocol connection establishment, Transport Layer Security (TLS) handshake, and overall request execution.

6. The HTTP client system based on phase tracing according to claim 1, characterized in that, The pluggable transport layer security protocol handshake component is configured to negotiate the application layer protocol negotiation protocol during the handshake process, and automatically select to use Hypertext Transfer Protocol version 1.1 or Hypertext Transfer Protocol version 2 to communicate with the server based on the negotiation result.

7. The HTTP client system based on phase tracing according to claim 1, characterized in that, The client engine also includes an automatic request header management module, which automatically detects and supplements necessary HTTP request header fields before sending a request.

8. A phase-tracing-based HTTP client request processing method, applied to the system as described in any one of claims 1 to 7, characterized in that, The method includes: Step 1, generating an HTTP request through the request builder and associating it with a tracker; Step 2, calling the tracker's request start callback through the tracker controller; Step 3, sequentially executing the following stages through the stage processor, with the tracker controller calling the corresponding tracking callback at the beginning and end of each stage: a) Domain name resolution stage, resolving the hostname in the HTTP request through the pluggable domain name resolver component; b) Transmission Control Protocol (TCP) connection stage, establishing a connection with the resolved or preset network address through the pluggable TCP connector component; c) Transport Layer Security (TLS) handshake stage, establishing a secure channel on the connection through the pluggable TLS handshake component when the HTTP request uses a security protocol; d) HTTP exchange stage, sending the HTTP request and receiving a response through the secure channel or the connection; Step 4, summarizing the performance data of each stage recorded by the tracker to generate a request execution report.

9. The HTTP client request processing method based on stage tracing according to claim 8, characterized in that, During the transmission control protocol connection phase, when there are multiple alternative network addresses, a parallel contention connection mechanism is adopted. The pluggable transmission control protocol connector component initiates multiple connection attempts in parallel, and cancels the remaining pending connection attempts after the first connection is successfully established.

Images