A distributed network proxy bypass request auditing system and method
By auditing requests at the traffic entry point of a microservice architecture, generating globally unique identifiers and asynchronously completing states, the intrusiveness and performance bottleneck issues of existing audit systems to downstream businesses are resolved, achieving efficient and reliable audit logging.
Patent Information
- Application Number
- CN202611300764.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-26
- Publication Date
- 2026-11-10
AI Technical Summary
Existing technologies for API request auditing in microservice architectures suffer from several drawbacks, including high invasiveness, high costs of cross-technology stack integration, performance bottlenecks, strong coupling between functions and business logic, high overhead of distributed deployment, and a lack of two-phase audit status completion mechanisms. These issues make it difficult to achieve efficient and reliable audit logging.
A distributed network proxy bypass request auditing system is adopted. By deploying a network proxy component at the traffic inlet, request pre-interception and response post-processing are performed to generate a globally unique audit identifier. The audit service module performs asynchronous status completion, and the log parsing and return module performs bypass streaming parsing to achieve transparent and asynchronous audit recording.
It reduces the intrusion on downstream business, improves the accuracy of the correlation between pre-request information and post-response results, reduces the risk of system cascading crashes, reduces deployment and maintenance costs, and ensures the integrity and reliability of audit logs.
Smart Images

Figure CN123075084A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer system development technology, and in particular to a distributed network proxy bypass request auditing system and method. Background Technology
[0002] In microservice architectures or distributed network clusters, complete audit logging of API (Application Programming Interface) requests is a fundamental requirement for system operation and maintenance, security compliance, and fault tracing. A complete audit log typically needs to include both pre-request information (such as request path, request method, anonymized business entity identifier, client IP, and request time) and post-response results (such as request processing result status identifier, response body size, and response time).
[0003] Existing technologies typically employ the following methods: (a) embedding audit logic within the business code of each microservice; (b) performing synchronous interception auditing at the API gateway layer; (c) assembling request link information by combining distributed tracing (such as cross-service association based on TraceId); and (d) deploying a lightweight proxy (Sidecar) alongside each microservice to send request and response information to a message queue for consumption by the audit service. This information is then combined with link logs from a full-text search engine and matched against preset audit rules to generate operation audit records. Among these, the design goal of tracing solutions is primarily performance analysis and fault location, rather than compliant storage of complete audit records; Sidecar-like solutions focus on filtering operations to be audited according to rule templates, and the number of proxy instances grows with the number of microservice instances. None of these solutions can rely solely on the access logs of the traffic entry gateway to complete a single-request audit loop of "pre-requisite information + response status," exhibiting the following drawbacks:
[0004] 1. Highly intrusive to existing business services and costly to integrate across technology stacks: Embedding logging in the business code of each microservice or registering global interceptors in the application layer framework both require the introduction of audit code in each business service. For existing services that are already in operation, the cost of transformation is extremely high; moreover, services with different technology stacks (such as Go, Java, Python, C++, etc.) need to develop and maintain their own audit logic, and the maintenance cost increases linearly with the number of services.
[0005] 2. Gateway-layer synchronous interception solutions suffer from performance bottlenecks and logical gaps: If unified synchronous interception is performed at the API gateway layer, the gateway middleware typically executes before request forwarding, at which point the backend service has not yet processed the request, and the middleware cannot obtain the final request processing result status. Forcibly postponing the audit logic to the response return stage for synchronous processing will cause synchronous blocking of the gateway's long-lived connections, significantly increasing the overall request latency and reducing the concurrency throughput performance of the core gateway.
[0006] 3. Strong coupling between auditing functions and core business functions leads to cascading failures: Traditional solutions run auditing logic and business logic in the same service process or synchronous call chain. When the underlying auditing storage service or auditing database experiences network fluctuations, write performance bottlenecks, or abnormal downtime, a cascading backlash effect occurs, directly blocking or hindering the normal response of core business requests and reducing the overall availability of the system.
[0007] 4. High overhead and heavy infrastructure dependency in distributed deployment scenarios: In Sidecar-like solutions, a proxy needs to be deployed next to each microservice instance. As the service scales up, the number of proxies increases accordingly, leading to higher operational and computing costs. Furthermore, these solutions typically rely on multiple infrastructure components such as message queues, distributed tracing, and full-text search engines to collaboratively assemble audit logs. In multi-gateway node deployment scenarios, further configuring high-performance log collection and real-time parsing components independently on each compute node will result in fragmented collection configurations, increasing the complexity of cluster operations and maintenance.
[0008] 5. Lack of a two-stage audit status completion mechanism for gateway scenarios: Existing solutions mostly generate audit events all at once after the proxy intercepts the response, or rely on link identifiers to retrieve logs from external systems and assemble them. They fail to address the contradiction of "the gateway cannot synchronously obtain the response status during the request phase, and it is not advisable to block request forwarding" by establishing a two-stage audit closed-loop mechanism: "pre-registering pending records during the request phase, and asynchronously completing the access logs into completed records during the response phase." When there is a delay in access log persistence and parsing, it is also difficult to effectively guarantee the integrity and eventual consistency of audit records. Summary of the Invention
[0009] This invention addresses the problems and shortcomings of existing technologies by providing a distributed network proxy bypass request auditing system and method.
[0010] The present invention solves the above-mentioned technical problems through the following technical solution:
[0011] This invention provides a distributed network proxy bypass request auditing system, characterized in that it includes a network proxy component, an audit service module, a storage medium, and a log parsing and return module. The network proxy component is deployed at the traffic entry point of the microservice cluster, and it integrates an audit interception unit and an access log output unit. The audit service module is an audit control center that is independent of the business service process and deployed in a spatially isolated manner, and provides an audit pre-registration interface and an audit status completion interface to the outside world.
[0012] When the original network request sent by the client arrives at the network proxy component, the audit interception unit is used to intercept the original network request in the control plane stage before the request is forwarded to the downstream business service, extract or generate request pre-registration information from the gateway context, and send the request pre-registration information to the audit service module through the network protocol to initiate audit pre-registration.
[0013] The audit service module receives the request information through the audit pre-registration interface, generates a globally unique audit identifier in the distributed cluster using a globally unique identifier algorithm, uses the audit identifier as the primary key, initializes and creates an initial audit log record with a status marked "pending completion" in the storage medium, writes the request information and the current system timestamp into the initial audit log record, and encapsulates the audit identifier in the protocol header field, returning it back to the network proxy component as a pre-registration response;
[0014] The network proxy component is used to extract the protocol header field carrying the audit identifier after receiving the pre-registration response, inject it into the request control domain that is about to be forwarded to the downstream business service, and the downstream business service does not need to modify the business code. The audit identifier is transparently transmitted along with the request control domain. The component also configures and activates the output policy of the access log output unit, which binds the audit identifier as a data column of the access log.
[0015] The network proxy component is also used to capture the response post-response result containing the request processing result status identifier returned by the downstream business service after processing the business at the end of the response lifecycle of the reverse proxy and return it to the client. At the same time, the access log output unit uses the closed-loop lifecycle characteristics of the reverse proxy to associate the audit identifier of the request phase with the response post-response result of the response phase in memory and write it as a complete log entry to the local access log file.
[0016] The log parsing and feedback module is used to incrementally monitor, stream filter and structure the access logs output by the network proxy component in a fully bypassed manner, extract audit identifiers and request processing result status identifiers, and asynchronously feedback them to the audit status completion interface of the audit service module.
[0017] The audit service module is used to retrieve the corresponding audit record in the storage medium based on the audit identifier in the data packet as the primary key, fill the corresponding audit record with the status identifier of the request processing result, and update the record status from "pending completion" to "completed", thus completing the full lifecycle audit loop.
[0018] This invention also provides a distributed network proxy bypass request auditing method, characterized in that it utilizes the aforementioned distributed network proxy bypass request auditing system, and the method includes the following steps:
[0019] Step S1: Request audit pre-interception and bypass pre-registration
[0020] When the original network request sent by the client arrives at the network proxy component, the audit interception unit intercepts the original network request in the control plane stage before the request is forwarded, extracts or generates request pre-requisite information from the gateway context, and sends the request pre-requisite information to the audit pre-registration interface of the audit service module through the network protocol;
[0021] Step S2: Unique audit identifier generation, record initialization, and reverse injection
[0022] After receiving the pre-registered request information, the audit service module performs the following atomic operations outside the business processing chain:
[0023] S2.1: Generate a globally unique audit identifier in the distributed cluster using a globally unique identifier algorithm;
[0024] S2.2: Using the audit identifier as the primary key, initialize and create an initial audit log record with the status marked "pending completion" in the storage medium, and write the request prerequisite information and the current system timestamp into the initial audit log record;
[0025] S2.3: Encapsulate the audit identifier in a protocol header field and return it back to the network proxy component as a pre-registration response;
[0026] Step S3: Merge and paste audit identifier relay pass-through with gateway access log fields onto disk.
[0027] After receiving the pre-registration response, the network proxy component performs two decoupled relay actions:
[0028] Action 1: Extract the protocol header field carrying the audit identifier and inject it into the request control domain that is about to be forwarded to the downstream business service. The downstream business service does not need to modify the business code, and the audit identifier is transparently transmitted along with the request control domain.
[0029] Action 2: Configure and activate the output policy of the access log output unit, and bind the audit identifier as a data column of the access log;
[0030] Step S4: Reverse response capture and gateway non-blocking disk write
[0031] After the downstream business service finishes processing the business, it returns a response post-response result containing the status identifier of the request processing result. The network proxy component captures the response post-response result at the end of the reverse proxy's response lifecycle and returns it to the client. At the same time, the access log output unit uses the closed-loop lifecycle feature of the reverse proxy to associate the audit identifier of the request phase with the response post-response result of the response phase in memory and writes it as a complete log entry into the local access log file.
[0032] Step S5: Bypass Streaming Log Collection and Structured Parsing
[0033] The log parsing and feedback module performs incremental monitoring, streaming filtering, and structured parsing of the access logs output by the network proxy component in a fully bypassed manner.
[0034] Step S6: Asynchronous completion and final persistence of audit log status
[0035] The log parsing and feedback module sends the parsed structured data objects to the audit status completion interface of the audit service module in a fully bypassed and asynchronous manner via network protocol or distributed message queue. The structured data objects include audit identifiers and request processing result status identifiers. The audit service module retrieves the corresponding audit record in the storage medium based on the audit identifier in the data packet as the primary key, fills the corresponding audit record with the request processing result status identifier, and updates the record status from "pending completion" to "completed", thus completing the full lifecycle audit loop.
[0036] In the field of request auditing for microservices and distributed cluster architectures, this invention can achieve the following technical effects:
[0037] 1. Achieves transparency of downstream business logic and reduces access overhead: This invention consolidates end-to-end request auditing capabilities into the control plane at the traffic entry point and the data plane in the bypass, achieving transparency of business logic for downstream heterogeneous business services. Regardless of the programming language or application layer development framework used by the backend business services, the business side does not need to modify the business code, thereby decoupling the deployment and access overhead of the auditing system from the number of business services and the types of technology stacks, reducing the implementation and subsequent maintenance costs of existing systems.
[0038] 2. This invention reconciles the conflicting relationships between gateway layer pre-interception and post-response status acquisition: It employs a two-stage correlation mechanism: asynchronous pre-registration of the request control plane, unilateral binding at the end of the reverse proxy lifecycle, and bypass streaming completion of access logs. This leverages the topological advantage of the network proxy component 100 in capturing request metadata during the pre-interception phase, while utilizing the access logs generated by the reverse proxy upon response as a bypass data source for the response results. This mechanism effectively improves the accuracy of the correlation between pre-request information and post-response results, reduces the probability of missing data across the entire link, and simultaneously ensures high concurrency throughput performance of the gateway layer's core routing data forwarding link.
[0039] 3. A process-level fault tolerance and disaster recovery boundary is established to reduce the risk of cascading crashes: In this invention, the traffic routing links of the core business are isolated and decoupled from subsequent audit record creation, asynchronous log collection, text parsing, and database persistence in terms of process space and computing resources. Any storage timeout, high-concurrency congestion, or abnormal fluctuations in the audit service module 200 or storage medium 210 are manifested as normal append writing of distributed node access logs. This avoids synchronous long-connection blocking of core request forwarding and client response at the network proxy component 100, thereby reducing the risk of system cascading crashes caused by traditional synchronous audit middleware and helping to ensure the overall availability of the business system.
[0040] 4. Reduced deployment and maintenance costs of the audit side in distributed cluster scenarios: In distributed deployment scenarios, this invention can uniformly aggregate the access logs generated by each network proxy component 100 instance to a shared storage domain, and centrally process them by the single-instance log parsing and return module 300, thereby reducing the need to repeatedly deploy collection and parsing components on each gateway node. This deployment method is a preferred implementation of this invention in distributed scenarios. Its core lies in the bypass streaming collection in step S5 and the asynchronous state completion mechanism in step S6, rather than a specific storage aggregation implementation method.
[0041] 5. Supports hot loading of audit policies and reduces the risk of log tampering through process and storage path isolation: This invention centralizes audit policy configuration in the network proxy component 100 and the log parsing and return module 300. Configuration files can be updated and take effect in real time through hot loading on the control plane, without requiring a restart of the downstream business service 500. Simultaneously, access logs are directly appended to an independent log file by the gateway process through the access log output unit 102. Log parsing and audit record persistence are performed in independent processes and storage media 210, respectively. Business service processes and business runtime memory spaces do not participate in log writing, thereby preventing the business side from tampering with the generated audit logs through conventional application interfaces, improving the integrity and traceability of audit data.
[0042] 6. Reduced dependence on Sidecar service-by-service deployment and tracing infrastructure: Compared to the solution of deploying a proxy next to each microservice and combining distributed tracing and full-text search engine to assemble audit events, this invention converges the single request audit loop to the access log and bypass link completion of the traffic entry gateway, and can complete the audit record from "to be completed" to "completed" status without relying on tracing identifiers and external search engines. Attached Figure Description
[0043] Figure 1 This is a schematic diagram of the overall system architecture provided in the embodiments of the present invention; it illustrates the topological relationship between the traffic entry layer, the audit isolation layer, the network shared storage component 400 in a distributed scenario, and the central physical storage address 700.
[0044] Figure 2 This is a flowchart illustrating the pre-registration stage of the audit request provided in an embodiment of the present invention; corresponding to steps S1 to S3, it shows the generation of audit identifiers, record initialization, and reverse injection process.
[0045] Figure 3 This is a flowchart illustrating the audit record status completion stage provided in this embodiment of the invention; corresponding to steps S4 to S7, it demonstrates the access log disk writing, bypass collection, asynchronous completion, and reconciliation mechanism.
[0046] Figure 4 This is a timing diagram showing the transmission of audit identifiers between components according to embodiments of the present invention; it illustrates the timing interactions between client 600, network proxy component 100, audit service module 200, downstream business service 500, and log parsing and feedback module 300.
[0047] Figure 5 This is a schematic diagram of the distributed log centralized shared storage and single-instance collection process provided in an embodiment of the present invention; it shows the architecture in which logs from multiple computing nodes are aggregated by the network shared storage component 400 and then centrally collected and parsed by the single-instance log parsing and backhaul module 300.
[0048] 100 — Network proxy component (including 101 audit interception unit and 102 access log output unit);
[0049] 200 — Audit Services Module;
[0050] 210 — Storage medium;
[0051] 300 — Log parsing and feedback module;
[0052] 400 — Network shared storage component;
[0053] 500 — Downstream business services;
[0054] 600 — Client;
[0055] 700 — Central physical storage address. Detailed Implementation
[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0057] like Figure 1-5 As shown, this embodiment of the invention provides a distributed network proxy bypass request auditing system, which adopts an architecture of "unified traffic entry proxy + audit service control plane + bypass completion of access log data plane". At the traffic entry point of the microservice cluster, the network proxy component 100 uniformly performs audit pre-registration and access log writing to disk, eliminating the need to deploy independent proxies next to each microservice. The system maintains the state transition of audit records from "pending completion" to "completed" through the audit service module 200, and bypasses and parses access logs through the log parsing and feedback module 300 to complete the asynchronous completion of response post-response results.
[0058] The distributed network proxy bypass request auditing system of this invention consists of the following spatially isolated and decoupled physical or logical modules, specifically including a network proxy component 100, an audit service module 200, a storage medium 210, a log parsing and return module 300, and a network shared storage component 400 containing a central physical storage address 700.
[0059] Network Proxy Component 100: Deployed as the traffic entry point in the microservice cluster, it can be deployed independently or distributed across multiple computing nodes. It is used for unified routing, forwarding, and interception control of client requests. Network Proxy Component 100 integrates an audit interception unit 101 and an access log output unit 102. Before a client request is forwarded to the backend business service (downstream business service), the audit interception unit 101 intercepts the request in the control plane phase and initiates audit pre-registration with the audit service module 200. At the end of the request-response loop, the access log output unit 102 merges the request context, response metadata, and audit identifier into a single access log entry, thus achieving pre-registration of single-request audit records on the gateway side and binding of post-response results.
[0060] Audit Service Module 200: An audit control center deployed independently of business service processes and in spatial isolation, providing an audit pre-registration interface and an audit status completion interface. Audit Service Module 200 receives request pre-registration information and creates an audit record with a status of "Pending Completion" before request forwarding via the audit pre-registration interface. Audit Service Module 200 receives response status information obtained through bypass parsing via the audit status completion interface, updates the corresponding audit record to "Completed," and persists the audit record containing request pre-registration information and response post-registration results to storage medium 210.
[0061] Log parsing and feedback module 300: An independent, bypass streaming log parsing program running separately from the business data plane. It incrementally monitors, streams, and performs structured parsing on the access logs output by the network proxy component 100, extracting audit identifiers and request processing result status identifiers, and asynchronously feedbacks them to the audit status completion interface of the audit service module 200. In a single-node deployment scenario, the log parsing and feedback module 300 directly monitors local access logs; in a distributed multi-node scenario, the log parsing and feedback module 300 can be centrally deployed on a designated central computing node to perform unified bypass processing on the aggregated multi-instance gateway access logs.
[0062] Network shared storage component 400: Deployed on the storage plane of the distributed cluster, it asynchronously and in real-time aggregates and mounts local access logs generated by multiple network proxy component 100 instances to the central physical storage address 700 of the network shared storage component 400 via the network shared storage protocol, for collection and parsing by the centrally deployed log parsing and backhaul module 300. The network shared storage component 400 is only a preferred implementation in distributed multi-gateway node scenarios and is not a necessary module for establishing the audit closed loop of this invention.
[0063] Storage medium 210: A persistent storage medium used for structured storage of audit pipeline data. It is directly connected to the audit service module 200 for read and write control and is used to store at least the audit identifier, request pre-requisite information, request processing result status identifier, and "pending completion" and "completed" status markers.
[0064] The working principle of the distributed network proxy bypass request auditing system is explained in detail below:
[0065] When the original network request (HTTP request) sent by client 600 arrives at network proxy component 100, audit interception unit 101 is used to intercept the original network request in the control plane stage before the request is forwarded to downstream business services, extract or generate request pre-registration information from the gateway context, and send the request pre-registration information to audit service module 200 through network protocol to initiate audit pre-registration.
[0066] The audit service module 200 is used to receive the request pre-registration information through the audit pre-registration interface, generate a globally unique audit identifier in the distributed cluster using a globally unique identifier algorithm, use the audit identifier as the primary key, initialize and create an initial audit log record with the status marked "pending completion" in the storage medium 210, write the request pre-registration information and the current system timestamp into the initial audit log record, and encapsulate the audit identifier in the protocol header field and return it back to the network proxy component 100 as a pre-registration response.
[0067] The network proxy component 100 is used to extract the protocol header field carrying the audit identifier after receiving the pre-registration response, inject it into the request control field that will be forwarded to the downstream business service 500, and the downstream business service 500 does not need to modify the business code, as the audit identifier is transparently transmitted along with the request control field; and to configure and activate the output policy of the access log output unit 102, which binds the audit identifier as a data column of the access log.
[0068] The network proxy component 100 is also used to capture the response post-response result containing the request processing result status identifier returned by the downstream business service 500 after processing the business at the end of the response lifecycle of the reverse proxy and return it to the client 600. At the same time, the access log output unit 102 uses the closed-loop lifecycle characteristics of the reverse proxy to associate the audit identifier of the request phase with the response post-response result of the response phase in memory and write it as a complete log entry into the local access log file.
[0069] The log parsing and feedback module 300 is used to incrementally monitor, stream filter, and perform structured parsing on the access logs output by the network proxy component 100 in a fully bypassed manner. It extracts audit identifiers and request processing result status identifiers, and asynchronously feeds them back to the audit status completion interface of the audit service module 200. In a single-node deployment scenario, the log parsing and feedback module 300 directly monitors the local access log file. In a scenario where multiple network proxy components 100 are deployed in a distributed manner across multiple physical machines, the log parsing and feedback module 300 aggregates the access logs generated by each instance to a unified directory via the network shared storage component 400 and the central physical storage address 700. The log parsing and feedback module 300, deployed as a single instance on a designated central computing node, performs unified bypass processing on the aggregated multi-instance gateway access logs.
[0070] Regardless of whether local listening or shared storage aggregation is used, the log parsing and return module 300 is used to perform the following operations:
[0071] By utilizing a file pointer listening mechanism, incremental and non-blocking retrieval and collection of logs aggregated from multiple gateway instances is performed.
[0072] By using pre-configured streaming parsing rules for streaming filtering, noise in ordinary static resource access logs without audit identifiers is eliminated, and the audit identifiers and request processing result status identifiers in the corresponding log lines are extracted accurately and in a structured manner.
[0073] The extracted data is assembled into lightweight structured data objects.
[0074] The audit service module 200 is used to retrieve the corresponding audit record in the storage medium 210 based on the audit identifier in the data packet as the primary key, fill the status identifier of the request processing result back into the corresponding audit record, and update the record status from "pending completion" to "completed", thus completing the full lifecycle audit closed loop.
[0075] In addition, when asynchronous transmission is interrupted due to network fluctuations, the log parsing and transmission module 300 enables the local location file recording mechanism to automatically resume transmission and retry after the network is restored.
[0076] If there is a physical delay in the writing and parsing of distributed access logs, causing an audit status query for the audit identifier to be initiated to the audit service module before the log parsing is completed, the periodic reconciliation scanner running inside the audit service module 200 is used to actively trace and backfill the status of abnormal records that are still in the "pending completion" state after exceeding the preset time threshold by reading the original log backup file in the network shared storage component 400 or the central physical storage address 700, thereby reducing the probability of audit status loss.
[0077] This invention also provides a distributed network proxy bypass request auditing method, which utilizes the aforementioned distributed network proxy bypass request auditing system. The specific steps of the method are as follows:
[0078] Step S1: Request audit pre-interception and bypass pre-registration
[0079] When the original network request sent by client 600 arrives at network proxy component 100, audit interception unit 101 intercepts the original network request in the control plane stage before request forwarding, extracts or generates request pre-requisite information from the gateway context, and sends the request pre-requisite information to the audit pre-registration interface of audit service module 200 through network protocol.
[0080] Step S2: Unique audit identifier generation, record initialization, and reverse injection
[0081] After receiving the pre-registered request information, the audit service module 200 performs the following atomic operations outside the business processing chain:
[0082] S2.1: Generate a globally unique audit identifier (AuditID / UUID) in the distributed cluster using a globally unique identifier algorithm.
[0083] S2.2: Using the audit identifier as the primary key, initialize and create an initial audit log record with the status marked "pending completion" in the storage medium 210, and write the request prerequisite information and the current system timestamp into the initial audit log record;
[0084] S2.3: Encapsulate the audit identifier in a protocol header field and return it back to the network proxy component 100 as a pre-registration response.
[0085] Step S3: Merge and paste audit identifier relay pass-through with gateway access log fields onto disk.
[0086] After receiving the pre-registration response, network proxy component 100 performs two decoupled relay actions:
[0087] Action 1: Extract the protocol header field carrying the audit identifier and inject it into the request control field that will be forwarded to the downstream business service 500. The downstream business service 500 does not need to modify the business code, and the audit identifier is transparently transmitted along with the request control field.
[0088] Action 2: Configure and activate the output policy of the access log output unit 102, and bind the audit identifier as a data column of the access log.
[0089] The two actions described above are executed decoupled within the gateway. Action 2 only configures the log field binding strategy and does not depend on the response results of downstream business services in Action 1.
[0090] Step S4: Reverse response capture and gateway non-blocking disk write
[0091] After processing the business, downstream business service 500 returns a response post-result containing the request processing result status identifier. Network proxy component 100 captures this response post-result at the end of the reverse proxy's response lifecycle and returns it to the client. At the same time, access log output unit 102 utilizes the closed-loop lifecycle characteristics of the reverse proxy to associate the audit identifier of the request phase with the response post-result of the response phase in memory and writes it as a complete log entry to the local access log file. The entire gateway layer interception and disk persistence logic is in a non-blocking state for data forwarding of core business.
[0092] Step S5: Bypass Streaming Log Collection and Structured Parsing
[0093] The log parsing and return module 300 performs incremental monitoring, streaming filtering, and structured parsing of the access logs output by the network proxy component 100 in a fully bypassed manner. In a single-node deployment scenario, the log parsing and return module 300 directly monitors the local access log file; in a scenario where multiple network proxy components 100 are deployed in a distributed manner across multiple physical machines, the access logs generated by each instance are aggregated to a unified directory through the network shared storage component 400 and the central physical storage address 700, and are centrally processed by the log parsing and return module 300, which is deployed as a single instance on a designated central computing node.
[0094] Regardless of whether local listening or shared storage aggregation is used, the log parsing and return module 300 performs the following operations:
[0095] S5.1: Utilizes a file pointer listening mechanism to perform incremental, non-blocking retrieval and collection of logs aggregated from multiple gateway instances;
[0096] S5.2: Streaming filtering is performed using pre-configured streaming parsing rules to eliminate noise in ordinary static resource access logs that do not contain audit identifiers, and to accurately and structurally extract the audit identifiers and request processing result status identifiers from the corresponding log lines.
[0097] S5.3: Assemble the extracted data into lightweight structured data objects.
[0098] Step S6: Asynchronous completion and final persistence of audit log status
[0099] The log parsing and feedback module 300 sends the parsed structured data object to the audit status completion interface of the audit service module 200 in a fully bypassed and asynchronous manner via network protocol or distributed message queue. The audit service module 200 retrieves the corresponding audit record from the storage medium 210 based on the audit identifier in the data packet as the primary key, fills the corresponding audit record with the request processing result status identifier, and updates the record status from "pending completion" to "completed," thus completing the full lifecycle audit loop. The step of updating the status from "pending completion" to "completed" constitutes the key technical feature that distinguishes this invention from one-time event generation audit schemes.
[0100] Step S7: Resumable download fault tolerance and timing misalignment self-healing reconciliation mechanism
[0101] The system incorporates a multi-level fault tolerance mechanism to combat distributed network jitter:
[0102] Resumable interruption: When the asynchronous transmission in step S6 is interrupted due to network fluctuations, the log parsing and transmission module 300 enables the local location file recording mechanism and automatically retryes the resumed transmission after the network is restored.
[0103] Time-sequence misalignment self-healing: If there is a physical delay in the writing and parsing of distributed access logs, causing an audit status query for the audit identifier to be initiated to the audit service module 200 before the log parsing is completed, the periodic reconciliation scanner running inside the audit service module 200 will actively trace back and fill in the status for abnormal records that are still in the "pending completion" state after exceeding the preset time threshold by reading the original log backup file in the network shared storage component 400 or the central physical storage address 700, thereby reducing the probability of audit status loss.
[0104] In this embodiment, the protocol header fields are represented as standard text HTTP headers under HTTP / 1.1 and HTTP / 1.0 protocols; and as binary flow control fields (Metadata) under HTTP / 2, HTTP / 3, or gRPC protocols.
[0105] In this embodiment, the network proxy component 100 can be an API gateway (such as Traefik, Kong, APISIX), a reverse proxy server (such as Nginx, HAProxy), or a sidecar proxy deployed on the data plane in a service mesh architecture (such as Envoy).
[0106] In this embodiment, the log parsing and feedback module 300 can use a general streaming log collection agent (such as Fluent Bit, Filebeat, or Logstash) or a custom resident script. The network shared storage component 400 can be mounted using NFS or SMB / CIFS, or it can be replaced with a persistent storage volume in a cloud-native environment. The above-mentioned shared storage aggregation method is only used for centralized log management in distributed deployment scenarios and does not change the core process of this invention, which involves gateway pre-registration and access log bypass completion.
[0107] In this embodiment, in step S6, the asynchronous communication between the log parsing and return module 300 and the audit service module 200 can, in addition to directly using HTTP / HTTPS / gRPC network protocols, also introduce a distributed message queue (such as Kafka, RabbitMQ, RocketMQ) as an intermediate decoupling buffer layer. Introducing the message queue is only used for peak smoothing and buffering of the return link and does not change the technical process of "asynchronously completing the audit record status after parsing the access log" in this invention.
[0108] The core difference between this invention and existing Sidecar + Link Tracing + Rule Matching auditing solutions lies in:
[0109] Audit pre-registration and access log writing are uniformly performed at the traffic entry gateway, eliminating the need to deploy a proxy next to each microservice;
[0110] During the request phase, an audit record with a status of "pending completion" is created. During the response phase, the gateway binds the audit identifier and response status to the same access log (access log, also called access log), and asynchronously completes it to "completed" through bypass parsing.
[0111] To address the technical contradiction that gateways cannot synchronously obtain response status when intercepting requests, and that blocking forwarding is not advisable, a two-stage mechanism of "control plane pre-registration + data plane log completion" is adopted to complete the audit loop.
[0112] The following describes a preferred embodiment of the present invention using a typical deployment of a residential energy storage cloud platform (HESS). Those skilled in the art should understand that the specific product names, interface paths, field names, and deployment parameters in the following embodiments are only used to illustrate the implementation process of the technical solution of the present invention and do not constitute a limitation on the scope of protection of the present invention. In the embodiments, the audit identifier is represented by audit_id.
[0113] Example 1: Single-Node Request Auditing Based on Traefik Gateway
[0114] 1. Deployment Architecture
[0115] This embodiment illustrates a preferred implementation of the present invention: pre-registration and access log persistence are uniformly completed by the traffic ingress Traefik gateway. In this embodiment, the network proxy component 100 adopts the Traefik reverse proxy / edge routing gateway and is deployed at the external API traffic ingress of the HESS cloud platform; the audit service module 200 is deployed as an independent microservice in an independent namespace of the Kubernetes cluster, isolated from the business process space of the downstream business service 500; the storage medium 210 adopts a relational database; the log parsing and return module 300 is deployed as a single-instance container within the same cluster, and accesses the access log files output by Traefik through container volume mounting or network file system with the node where Traefik is located.
[0116] 2. Request pre-registration (corresponding to steps S1 to S3)
[0117] When client 600 initiates an HTTPS request to the HESS cloud platform and reaches Traefik, Traefik's built-in audit interception unit 101 extracts the following pre-request information before forwarding the request to the downstream business service 500: the request source IP address; the request method (GET / POST / PUT / DELETE, etc.); the request URI path; the timestamp of the request reaching the gateway; and optional user identification (such as the user ID in JWT, if it is already carried in the request header).
[0118] The audit interception unit 101 encapsulates the above information into a JSON request body and calls the pre-registration interface of the audit service module 200 via HTTP POST, for example:
[0119] POST / api / v1 / audit / pre-register
[0120] After receiving the pre-registration request, the audit service module 200 performs the atomic operation described in step S2: generating a globally unique audit identifier audit_id using the UUIDv4 algorithm; creating an audit log record with a status of "pending completion" in the storage medium 210 using audit_id as the primary key, and writing the request pre-requisite information and the current system timestamp; then encapsulating audit_id in the HTTP response header X-Audit-Id and returning it to Traefik.
[0121] After receiving the pre-registration response, Traefik executes step S3:
[0122] Action 1: Inject the X-Audit-Id into the 500 request header of the downstream business service. The downstream microservice can transparently pass through this identifier without modifying the business code.
[0123] Action 2: Configure X-Audit-Id as an independent field column in the access log through Traefik's access log output unit 102, for example, the log format template contains the placeholder %{X-Audit-Id}i.
[0124] 3. Access log writing to disk and bypass completion (corresponding to steps S4 to S6)
[0125] After the downstream business service completes its business processing (500), Traefik returns a 600 response to the client at the end of the reverse proxy response lifecycle. At this time, Traefik access log output unit 102 simultaneously writes the following to a single log record: the audit_id from the request phase; the HTTP status code (i.e., the request processing result status identifier) from the response phase; the response time (milliseconds); and basic fields such as the request method, URI, and client IP.
[0126] Log entry example (fields separated by vertical bars):
[0127] 2026-07-09T10:15:32+08:00 | 192.168.1.100 | POST | / api / hess / device / status | audit_id=a3f8c2e1-... | status=200 | duration=45ms
[0128] The log parsing and feedback module 300 incrementally monitors the aforementioned access log files in a bypass manner (using inotify or file pointer offset mechanism), filters static resource access records that do not contain audit_id using streaming parsing rules, extracts the audit_id and status fields, and assembles them into a lightweight JSON object.
[0129] {"audit_id":"a3f8c2e1-...","http_status":200,"completed_at":"2026-07-09T10:15:32+08:00"}
[0130] Then, the status completion interface of the audit service module 200 is asynchronously submitted via HTTP POST, for example: POST / api / v1 / audit / access-ingest.
[0131] The audit service module 200 retrieves the corresponding record in the storage medium 210 based on the audit_id, fills in the HTTP status code, and updates the record status from "pending completion" to "completed".
[0132] 4. Fault tolerance mechanism (corresponding to step S7)
[0133] The log parsing and return module 300 maintains a local position file (such as offset.checkpoint) and records the last file offset that has been successfully parsed and returned. When the asynchronous return in step S6 is interrupted due to network jitter, the module continues to read from the checkpoint position after the network is restored, thus realizing breakpoint resume.
[0134] The audit service module 200 internally runs a periodic reconciliation scanner, which scans abnormal records that are still in the "pending completion" state and whose creation time exceeds 10 minutes every 5 minutes by default. It actively reads the network shared storage component 400 or local log backup files for traceback and backfilling to reduce the probability of the audit status being suspended for a long time due to log disk delays.
[0135] Example 2: Equivalent Example Based on Nginx Gateway
[0136] This embodiment follows the same technical process as Embodiment 1, the only difference being that the network proxy component 100 uses an Nginx reverse proxy.
[0137] Audit interception unit 101 calls the audit service module 200 pre-register interface before request forwarding via Nginx access_by_lua or ngx_http_lua_module;
[0138] The audit_id returned during pre-registration is injected into the downstream request header via ngx.req.set_header();
[0139] Access log output unit 102 uses the custom format log_format to write $http_x_audit_id and $status to the same access.log entry.
[0140] The interaction process of the log parsing and return module 300, the audit service module 200, and the storage medium 210 is consistent with that in Embodiment 1, proving that the present invention does not depend on a specific gateway product and is applicable to various reverse proxy implementations.
[0141] Example 3: Multi-node distributed deployment example of HESS cloud platform
[0142] This embodiment illustrates a preferred deployment method of the present invention in a distributed cluster, used to explain the implementation of multi-node log aggregation and single-instance collection, and is not a necessary technical feature of the method of the present invention.
[0143] In the HESS cloud platform production environment, the network proxy component 100 is deployed in a distributed manner across three physical compute nodes (compute nodes A, B, ..., N) as a Traefik instance, and each node independently outputs access log files:
[0144] access_node1.log
[0145] access_node2.log
[0146] access_nodeN.log
[0147] Each node mounts and outputs its local logs in real time to the central physical storage address 700 of the network shared storage component 400 via the NFS protocol. The log parsing and return module 300 is deployed as a single instance on the central physical node. It performs centralized bypass streaming collection and parsing of the aggregated multi-node logs using wildcard access_*.log, avoiding the duplication of the collection agent on each gateway node.
[0148] Except for the log aggregation method, the business logic of steps S1 to S7 is exactly the same as in Example 1. This deployment method decouples the cluster-side audit collection computing power overhead from the number of gateway nodes. When the gateway nodes are horizontally scaled, there is no need to synchronously expand the number of log collection instances.
[0149] Example 4: Extended Example of Decoupling Backhaul Using Message Queues
[0150] This embodiment illustrates an extended implementation of the backhaul link in step S6. Based on embodiment one or three, when the backhaul concurrency pressure is high, the communication between the log parsing backhaul module 300 and the audit service module 200 in step S6 can be further enhanced by introducing a Kafka message queue as an intermediate buffer layer.
[0151] The log parsing and feedback module 300 publishes the parsed structured data objects to a Kafka Topic (such as hess-audit-ingest).
[0152] The audit service module 200 asynchronously subscribes to the Topic as a consumer group, performing state completion and persistence.
[0153] This method provides additional peak-shaving buffering capability when the audit service module 200 is temporarily unavailable or during high-concurrency writes. Its essential technical process is still the "bypass acquisition → asynchronous backhaul → status completion" described in this invention, with only the backhaul communication medium changing.
[0154] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.
Claims
1. A distributed network proxy bypass request auditing system, characterized in that, It includes a network proxy component, an audit service module, storage media, and a log parsing and return module. The network proxy component is deployed at the traffic entry point of the microservice cluster. It integrates an audit interception unit and an access log output unit. The audit service module is an audit control center that is independent of the business service process and spatially isolated. It provides an audit pre-registration interface and an audit status completion interface to the outside world. When the original network request sent by the client arrives at the network proxy component, the audit interception unit is used to intercept the original network request in the control plane stage before the request is forwarded to the downstream business service, extract or generate request pre-registration information from the gateway context, and send the request pre-registration information to the audit service module through the network protocol to initiate audit pre-registration. The audit service module receives the request information through the audit pre-registration interface, generates a globally unique audit identifier in the distributed cluster using a globally unique identifier algorithm, uses the audit identifier as the primary key, initializes and creates an initial audit log record with a status marked "pending completion" in the storage medium, writes the request information and the current system timestamp into the initial audit log record, and encapsulates the audit identifier in the protocol header field, returning it back to the network proxy component as a pre-registration response; The network proxy component is used to extract the protocol header field carrying the audit identifier after receiving the pre-registration response, inject it into the request control domain that is about to be forwarded to the downstream business service, and the downstream business service does not need to modify the business code. The audit identifier is transparently transmitted along with the request control domain. The component also configures and activates the output policy of the access log output unit, which binds the audit identifier as a data column of the access log. The network proxy component is also used to capture the response post-response result containing the request processing result status identifier returned by the downstream business service after processing the business at the end of the response lifecycle of the reverse proxy and return it to the client. At the same time, the access log output unit uses the closed-loop lifecycle characteristics of the reverse proxy to associate the audit identifier of the request phase with the response post-response result of the response phase in memory and write it as a complete log entry to the local access log file. The log parsing and feedback module is used to incrementally monitor, stream filter and structure the access logs output by the network proxy component in a fully bypassed manner, extract audit identifiers and request processing result status identifiers, and asynchronously feedback them to the audit status completion interface of the audit service module. The audit service module is used to retrieve the corresponding audit record in the storage medium based on the audit identifier in the data packet as the primary key, fill the corresponding audit record with the status identifier of the request processing result, and update the record status from "pending completion" to "completed", thus completing the full lifecycle audit loop.
2. The distributed network proxy bypass request auditing system as described in claim 1, characterized in that, The system also includes a network shared storage component containing a central physical storage address. The network shared storage component is deployed on the storage surface of the distributed cluster. Through the network shared storage protocol, the local access logs generated by multiple network proxy component instances are asynchronously and in real time aggregated and mounted to the central physical storage address for collection and parsing by the centrally deployed log parsing and backhaul module. In a single-node deployment scenario, the log parsing and return module is used to directly monitor local access log files; In scenarios where multiple network proxy components are deployed in a distributed manner across multiple physical machines, the log parsing and backhaul module is used to aggregate the access logs generated by each instance to a unified directory through the network shared storage component and the central physical storage address. The log parsing and backhaul module, which is deployed on a designated central computing node by a single instance, performs unified bypass processing on the aggregated multi-instance gateway access logs.
3. The distributed network proxy bypass request auditing system as described in claim 2, characterized in that, Regardless of whether local listening or shared storage aggregation is used, the log parsing and return module is used to perform the following operations: By utilizing a file pointer listening mechanism, incremental and non-blocking retrieval and collection of logs aggregated from multiple gateway instances is performed. By using pre-configured streaming parsing rules for streaming filtering, noise in ordinary static resource access logs without audit identifiers is eliminated, and the audit identifiers and request processing result status identifiers in the corresponding log lines are extracted accurately and in a structured manner. The extracted data is assembled into lightweight structured data objects.
4. The distributed network proxy bypass request auditing system as described in claim 2, characterized in that, When asynchronous transmission is interrupted due to network fluctuations, the log parsing and transmission module enables the local location file recording mechanism to automatically resume transmission and retry after the network is restored. If there is a physical delay in the writing and parsing of distributed access logs, causing an audit status query for the audit identifier to be initiated to the audit service module before the logs have been parsed, the periodic reconciliation scanner running inside the audit service module is used to proactively trace and backfill the status of abnormal records that are still in the "pending completion" state after exceeding a preset time threshold by reading the original log backup file in the network shared storage component or the central physical storage address, thereby reducing the probability of audit status loss.
5. A method for auditing bypass requests in a distributed network proxy, characterized in that, It is implemented using the distributed network proxy bypass request auditing system as described in any one of claims 1-4, and the method includes the following steps: Step S1: Request audit pre-interception and bypass pre-registration When the original network request sent by the client arrives at the network proxy component, the audit interception unit intercepts the original network request in the control plane stage before the request is forwarded, extracts or generates request pre-requisite information from the gateway context, and sends the request pre-requisite information to the audit pre-registration interface of the audit service module through the network protocol; Step S2: Unique audit identifier generation, record initialization, and reverse injection After receiving the pre-registered request information, the audit service module performs the following atomic operations outside the business processing chain: S2.1: Generate a globally unique audit identifier in the distributed cluster using a globally unique identifier algorithm; S2.2: Using the audit identifier as the primary key, initialize and create an initial audit log record with the status marked "pending completion" in the storage medium, and write the request prerequisite information and the current system timestamp into the initial audit log record; S2.3: Encapsulate the audit identifier in a protocol header field and return it back to the network proxy component as a pre-registration response; Step S3: Merge and paste audit identifier relay pass-through with gateway access log fields onto disk. After receiving the pre-registration response, the network proxy component performs two decoupled relay actions: Action 1: Extract the protocol header field carrying the audit identifier and inject it into the request control domain that is about to be forwarded to the downstream business service. The downstream business service does not need to modify the business code, and the audit identifier is transparently transmitted along with the request control domain. Action 2: Configure and activate the output policy of the access log output unit, and bind the audit identifier as a data column of the access log; Step S4: Reverse response capture and gateway non-blocking disk write After the downstream business service finishes processing the business, it returns a response post-response result containing the status identifier of the request processing result. The network proxy component captures the response post-response result at the end of the reverse proxy's response lifecycle and returns it to the client. At the same time, the access log output unit uses the closed-loop lifecycle feature of the reverse proxy to associate the audit identifier of the request phase with the response post-response result of the response phase in memory and writes it as a complete log entry into the local access log file. Step S5: Bypass Streaming Log Collection and Structured Parsing The log parsing and feedback module performs incremental monitoring, streaming filtering, and structured parsing of the access logs output by the network proxy component in a fully bypassed manner. Step S6: Asynchronous completion and final persistence of audit log status The log parsing and feedback module sends the parsed structured data objects to the audit status completion interface of the audit service module in a fully bypassed and asynchronous manner via network protocol or distributed message queue. The structured data objects include audit identifiers and request processing result status identifiers. The audit service module retrieves the corresponding audit record in the storage medium based on the audit identifier in the data packet as the primary key, fills the corresponding audit record with the request processing result status identifier, and updates the record status from "pending completion" to "completed", thus completing the full lifecycle audit loop.
6. The distributed network proxy bypass request auditing method as described in claim 5, characterized in that, In step S5, in a single-node deployment scenario, the log parsing and return module directly listens to the local access log file; in a scenario where multiple network proxy components are deployed in a distributed manner on multiple physical machines, the access logs generated by each instance are aggregated to a unified directory through the network shared storage component and the central physical storage address, and are centrally processed by the log parsing and return module deployed on a designated central computing node.
7. The distributed network proxy bypass request auditing method as described in claim 6, characterized in that, In step S5, regardless of whether local listening or shared storage aggregation is used, the log parsing and return module performs the following operations: S5.1: Utilizes a file pointer listening mechanism to perform incremental, non-blocking retrieval and collection of logs aggregated from multiple gateway instances; S5.2: Streaming filtering is performed using pre-configured streaming parsing rules to eliminate noise in ordinary static resource access logs that do not contain audit identifiers, and to accurately and structurally extract the audit identifiers and request processing result status identifiers from the corresponding log lines. S5.3: Assemble the extracted data into lightweight structured data objects.
8. The distributed network proxy bypass request auditing method as described in claim 6, characterized in that, Step S7: Resumable download fault tolerance and timing misalignment self-healing reconciliation mechanism The system incorporates a multi-level fault tolerance mechanism to combat distributed network jitter: Resumable interruption: When the asynchronous transmission in step S6 is interrupted due to network fluctuations, the log parsing and transmission module enables the local location file recording mechanism and automatically retry the resume interruption after the network is restored. Time-sequence misalignment self-healing: If there is a physical delay in the writing and parsing of distributed access logs, causing an audit status query for the audit identifier to be initiated to the audit service module before the log is parsed, the periodic reconciliation scanner running inside the audit service module will actively trace back and fill in the status for abnormal records that are still in the "pending completion" state after exceeding the preset time threshold by reading the original log backup file in the network shared storage component or the central physical storage address, thereby reducing the probability of audit status loss.
