Distributed log access method based on log framework and network communication framework

Abstract

The invention discloses a distributed log access method based on a log framework and a network communication framework. According to the method, a log event is captured by extending a log framework output component at a service application end, and filtering and desensitization processing is carried out; then aggregating and compressing the processed log events in batches; packaging the compressed data into a custom protocol format by managing a multi-connection network client, and transmitting the packaged data to a server; and after executing protocol analysis and decompression on the received data, the server distributes the log to a corresponding storage engine for storage according to a pre-configuration strategy. Full-link integrated management from log collection, processing, transmission to storage is achieved, log processing efficiency and system reliability are effectively improved through non-intrusive collection, batch compression transmission and reliable protocol analysis, resource consumption and operation and maintenance complexity are reduced at the same time, and user experience is improved. The defects that a traditional scheme is complex in architecture, poor in real-time performance and high in invasiveness are overcome.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer software, and particularly relates to a distributed log access method based on a log framework and a network communication framework. BACKGROUND

[0002] Under the current large-scale distributed system architecture, efficient log management is the core link to ensure system observability, fault diagnosis and business analysis. At present, the industry generally adopts the technology stack represented by ELK or EFK as a standard solution, which realizes log collection, transmission, storage and visualization by combining multiple independent components such as Elasticsearch, Logstash / Fluentd, Kibana, etc. However, such a solution has exposed significant defects in actual deployment and operation: first, its multi-component architecture leads to high system complexity, not only is the initial deployment cumbersome, but each component (such as the input, filtering, and output plugins of Logstash) requires a large number of configurations and is dependent on each other, making the operation difficulty and technical threshold high; second, the entire technology stack consumes a huge amount of server memory, CPU and other resources, especially when the log volume surges, the heavy resource overhead makes it difficult to apply in resource-constrained scenarios such as edge computing and low-cost projects; third, some collection solutions require deep coupling in the business code or code insertion through AOP, which is highly invasive to the business system, damaging the purity and maintainability of the code; finally, the file polling-based collection mechanism (such as Filebeat) has inherent delays and cannot achieve real-time log stream processing, making it difficult to meet the needs of scenarios such as financial transactions and online monitoring that require high real-time performance of logs. Therefore, there is an urgent need in the art for a distributed log access solution that is lightweight, low-invasive, high-performance and high-real-time to overcome the comprehensive bottlenecks of the above existing technologies. SUMMARY

[0003] (I) Purpose of the Invention In order to overcome the above shortcomings, the purpose of the present application is to provide a distributed log access method based on a log framework and a network communication framework, to solve the technical problems of high deployment and operation cost and poor real-time performance caused by the complex architecture and high resource consumption of existing distributed log solutions.

[0004] (II) Technical Solution To achieve the above purpose, the technical solution provided by the present application is as follows: A distributed log access method based on a log framework and a network communication framework, comprising the following steps: Step S1, at the business application end, capturing log events by extending the output component of the log framework; Step S2, filtering and desensitizing the captured log events; Step S3, batch aggregation of the processed log events, and compression of the batch log data after aggregation; Step S4, encapsulating the compressed batch log data into a custom protocol format by a network client managing multiple network connections, and transmitting the data to a log server; Step S5, protocol analysis and data decompression of the received data at the log server; Step S6, distribution of the decompressed log data to corresponding storage engines for storage according to a preconfigured distribution strategy.

[0005] The present application first realizes non-invasive capture of log events by extending the log framework component, ensuring the purity of business code from the source. The subsequent filtering and desensitization process completes preliminary data cleaning and security protection at the data export, improving the quality and compliance of log data. The key steps of batch aggregation and compression convert discrete log events into compact batch data packets, greatly optimizing network transmission efficiency. Through the management of multi-connection network clients and custom protocols for data transmission, not only the efficiency and reliability of the transmission process are guaranteed, but also the decoupling of the underlying communication details is achieved. The protocol analysis and decompression at the server ensure the complete restoration and correct interpretation of the data. Finally, based on the preconfigured strategy, intelligent distribution enables log data to be flexibly and accurately routed to the appropriate storage engine, meeting diverse storage and subsequent analysis needs. This series of steps work together to achieve a lightweight architecture, efficient processing, reliable transmission, and convenient operation of distributed log access, effectively overcoming the comprehensive defects of complex architecture, high resource consumption, poor real-time performance, and strong invasiveness in existing technologies.

[0006] In some embodiments, in step S3, the triggering condition for batch aggregation is reaching a preset log quantity threshold or reaching a preset time window threshold.

[0007] This dual-threshold triggering mechanism makes the log sending action no longer limited to a single condition, ensuring timely triggering of sending to reduce memory occupation during log generation peaks, and ensuring timely transmission of logs through time windows during low peaks, effectively balancing the relationship between processing real-time performance and system resource consumption. The system can adapt to different business load scenarios, ensuring log collection timeliness while avoiding network resource waste caused by frequent small batch transmission.

[0008] In some embodiments, in step S3, the batch log data is compressed using the Snappy algorithm.

[0009] The compression algorithm significantly reduces the volume of log data, thereby reducing the occupation of network transmission bandwidth, with high speed and reasonable compression rate. This processing enables more log information to be transmitted under the same network conditions, improving the data transmission efficiency of the entire system and providing technical feasibility for deployment in bandwidth-limited environments. Moreover, the compression and decompression process is fast, and the impact on system processing delay is minimal.

[0010] In some embodiments, in the step S4, the network client manages multiple network connections through a connection pool and selects a target server node for data transmission using a weighted round-robin strategy.

[0011] The use of a connection pool avoids the overhead of establishing a new connection for each log transmission, and the reuse of connections greatly improves network utilization efficiency. Combined with the weighted round-robin load balancing strategy, log traffic can be reasonably distributed according to the processing capacity of the server nodes, preventing overload of individual nodes and improving the throughput capacity and service reliability of the entire log service cluster, achieving high availability and scalability of the system.

[0012] In some embodiments, in the step S4, the custom protocol format is a binary protocol, and the encapsulated content includes a protocol header, a compressed log body, and a checksum; the protocol header contains log quantity, pre-compression size, and post-compression size information.

[0013] Compared with text protocols such as JSON, the protocol design of the present application has smaller data packet size and faster parsing speed. The built-in checksum mechanism provides protection for data integrity, and the explicit length information combined with length prefix-based frame decoding fundamentally solves the TCP sticking and unpacking problem, ensuring the reliability and correctness of network transmission.

[0014] In some embodiments, in the step S5, the protocol parsing process includes length prefix-based frame decoding to solve the TCP sticking problem.

[0015] By pre-reading and parsing the exact length of each independent data packet, the received raw byte stream can be accurately cut into complete and single logical messages, thereby fundamentally avoiding the sticking and unpacking problems caused by the lack of message boundary characteristics in the TCP protocol. Even if multiple data packets are merged or split during network transmission, the server can always correctly restore the original data unit from the sender, ensuring the integrity and consistency of log data at the transmission level. This improves the accuracy and robustness of protocol parsing, simplifies the data processing logic of the server, and improves the stability and reliability of subsequent decompression, verification, and distribution steps, which is an indispensable key link to ensure the high reliability of the entire log transmission channel.

[0016] In some embodiments, after the step S4, the method further comprises a step S4a: if the log data transmission fails or times out, retrying; if the retrying still fails, writing the log data into a local cache file.

[0017] This set of failure handling and degradation mechanism constitutes the security line of the system. The retry mechanism deals with short-term network fluctuations, and the local file cache provides a bottom solution for persistent failure, ensuring that log data will not be lost under any abnormal circumstances. The use of memory-mapped files further improves the read-write performance of the local cache, laying a solid foundation for subsequent network recovery and data retransmission.

[0018] In some embodiments, the local cache file is stored in a memory-mapped file.

[0019] In some embodiments, in the step S4, the network client and the log server maintain connection activity through a heartbeat detection mechanism.

[0020] Periodic heartbeat packets can timely discover and clean up invalid network connections, maintaining the health status of connections in the connection pool. It effectively avoids the "half-open connection" problem caused by network intermediate equipment timeout disconnection or server accidental termination, ensuring the long-term availability and stability of the log transmission channel, and reducing the sending failures caused by connection problems.

[0021] In some embodiments, the method is further configured and monitored through a management console, the configuration including setting at least one of log filtering rules, desensitization rules, batch size, compression switch and storage routing strategy; the monitoring includes monitoring at least one of log traffic, transmission delay and error rate.

[0022] The configuration function covers the whole link strategy setting of log collection, transmission and storage, and the monitoring function real-time tracks the key operation indicators of the system. This centralized operation and maintenance management mode greatly simplifies the maintenance work of the system, enables the operation and maintenance personnel to quickly adjust the system behavior and timely discover potential problems, thereby improving the operability and transparency of the entire log system. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is the overall system architecture diagram of the distributed log access method based on the log framework and the network communication framework of the present application; Figure 2 is the log collection and sending process diagram in the client in the distributed log access method based on the log framework and the network communication framework of the present application; Figure 3 is the log receiving and storage process flowchart in the server in the distributed log access method based on the log framework and the network communication framework of the present application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0025] The present invention provides a distributed log access method based on a log framework and a network communication framework. Its core lies in building a complete technical solution from log collection to storage by deeply integrating these two frameworks.

[0026] S1. This method first captures log events on the business application side by extending the Logback framework's Appender component. This customized Logback Appender is integrated into the business application's logging configuration, thereby seamlessly intercepting all output log events.

[0027] S2. Further, the Appender performs filtering and desensitization processing on the captured log events; the filtering operation can be performed according to the preset log level, while the desensitization replaces sensitive information such as mobile phone numbers and email addresses according to the configured rules.

[0028] S3. Based on this, the processed log events are batch aggregated. This aggregation process takes place in an in-memory queue. When the number of logs in the queue reaches a preset threshold or a preset time window, subsequent processing is triggered. In particular, after aggregation, the aggregated log data is compressed using the Snappy compression algorithm, which aims to significantly reduce the amount of data transmitted over the network.

[0029] S4. Subsequently, a Netty client managing multiple network connections encapsulates the compressed batch log data into a specific custom binary protocol format and transmits it to a remote log server. This Netty client uses a connection pool to manage these network connections and employs load balancing strategies such as weighted round-robin to select an available connection from the pool for data transmission. It is worth noting that the encapsulated content of the custom binary protocol format explicitly includes a protocol header, the compressed log body, and a checksum field; the protocol header itself contains key information such as the number of logs, the size of the data before compression, and the size of the data after compression.

[0030] S5, in the log service end, data is received by the Netty service end instance, and protocol analysis and data decompression are performed on the received data. The protocol analysis process is particularly important, which includes a length prefix-based frame decoding step, so as to solve the inherent stick packet / dismantling packet problem in TCP transmission, and to ensure that an independent and complete data packet can be accurately restored. After successful analysis and verification of data integrity, the service end uses the Snappy algorithm to decompress the log data.

[0031] S6, finally, the log distributor built in the service end distributes the decompressed original log data to the target storage engine for persistent storage, such as Kafka, Elasticsearch or HDFS, through the corresponding storage engine adapter according to the distribution strategy preconfigured through the management console.

[0032] The embodiments are described in detail below with reference to the accompanying drawings Figure 1 The overall system architecture of the distributed log access implementation method of the application is schematically shown. The architecture is divided into a business application layer, a log collection layer, a log service layer and a storage layer from top to bottom, and is supplemented by a unified management console. The business application layer includes a plurality of independent business application instances, such as order service, payment service, etc. The Logback Appender customized by the present application is integrated in each application instance. The log collection layer is a set of components deployed in each application instance, and the core includes the customized Logback Appender and the Netty client connection pool. The Appender is responsible for capturing, formatting, filtering, desensitizing, batch aggregating and compressing log events. The Netty client connection pool is responsible for maintaining network connections to multiple nodes of the log service layer, and selecting connections for data transmission according to a load balancing strategy. The log service layer is composed of a cluster of multiple Netty service end instances, each service end instance including a protocol decoder, a log distributor and a plurality of storage engine adapters. The protocol decoder is responsible for analyzing the custom binary protocol and verifying data; the log distributor directs the log to different storage engine adapters according to the configured routing strategy; and the adapter is responsible for converting the log data into the format required by the target storage engine and performing writing. The storage layer is composed of various storage engine clusters, such as Kafka cluster, Elasticsearch cluster and HDFS cluster, for the final persistence of logs. The management console serves as the operation and maintenance hub of the system, and provides an interface for configuring log collection, transmission, storage strategy and monitoring system running indicators.

[0033] Figure 2The collection and sending process of logs on the client side is described in detail. The process starts with a business thread calling the Logback interface to output a log event. The event is captured by a customized Logback Appender. Then, the Appender filters and sanitizes the log according to the configuration rules obtained from the management console. The processed log event is placed in an asynchronous in-memory batch queue for temporary storage. When the number of logs in the queue reaches the set batch size or the waiting time reaches the set time window, batch processing is triggered. The accumulated batch log data is sent to the compression module for compression using the Snappy algorithm. The compressed data and the protocol header containing information such as the number of logs, the size before and after compression, and the checksum are packaged into a custom binary protocol packet. The Netty client connection pool selects a healthy and available network connection according to the load balancing strategy (such as weighted round robin) and sends the packet asynchronously to the log server. The client waits for the response from the server. If a successful response is received, the corresponding local buffer is cleared; if the sending fails or times out, S4a: retries a limited number of times. In particular, if the network is unavailable or multiple retries fail, as a degradation guarantee, this batch of log data will be written to a local cache file. The file preferably uses the memory-mapped file technology to achieve high-performance read and write. When the network is restored, the system will automatically resend the logs temporarily stored in the local file, ensuring that data is not lost.

[0034] Figure 3 The receiving and storage process of logs on the server side is then shown. The Netty server receives data packets from the client. The protocol parsing module then parses the data packet. First, frame decoding is performed according to the length prefix to solve the problem of sticky packet, then the protocol header information is parsed, and the checksum is used to verify data integrity. If the verification fails, a failure response is returned to the client, prompting the client to resend. If the verification passes, the Snappy algorithm is used to decompress the log body, restoring the original batch log data. The decompressed log data is submitted to the log distributor, which distributes the log to the corresponding storage engine adapter, such as the Kafka adapter, Elasticsearch adapter, etc., according to the routing strategy preset by the management console, such as by application name or log type. The adapter is responsible for converting the log data into the format required by the target storage engine and performing the write operation. Finally, the write result is returned to the client, completing a complete request-response interaction.

[0035] It needs to be particularly pointed out that if Logback is simply combined with Netty, a stable, efficient and production environment log collection and transmission scheme cannot be directly obtained. If only surface integration is performed by those skilled in the art, for example, directly calling the synchronous sending method of Netty in the Logback Appender, a series of serious technical challenges will be faced, which highlights the necessity and non-obviousness of the deep integration of the present application. The main difficulty lies in: Performance and stability contradiction: the Appender of Logback is executed synchronously in the business thread by default, if the network I / O operation is directly coupled, the business thread is easily blocked when the network jitter or server delay occurs, resulting in a sharp decline in application performance and even service unavailability.

[0036] Reliability guarantee is complex: the network environment is naturally unstable, simple integration cannot handle network interruption, server downtime and other abnormal situations, which easily leads to log loss. To realize reliable transmission, a complete retry, local degradation and offline retransmission mechanism must be designed.

[0037] Resource management and efficiency optimization: if high-frequency log output corresponds to frequent network requests, a large number of small data packets will be generated, resulting in low network utilization and huge server pressure. Batch aggregation, data compression, connection multiplexing and other optimization strategies must be introduced, which puts higher requirements on memory management and thread model.

[0038] Protocol and communication detail processing: the problem of TCP stick packet / dismantling must be properly solved, otherwise the log data cannot be correctly parsed. Custom efficient and compact binary protocol and reliable codec need in-depth network programming knowledge.

[0039] The present application solves the performance blocking problem through asynchronous processing and memory queue; builds reliability guarantee through retry mechanism and local MMAP file cache; greatly improves transmission efficiency through batch aggregation, compression algorithm and connection pool; ensures the reliability of communication through custom binary protocol and length prefix based frame decoding. These synergistic technical means jointly constitute a complete solution, successfully integrate the log collection capability of Logback and the network communication capability of Netty, overcome the inherent defects brought by simple combination, and finally achieve the technical effects of lightweight, high performance and high availability.

[0040] It should be understood that the foregoing detailed description of the application, rather than limiting the application, is intended to explain and describe the current implementation of the application. Therefore, any modification, equivalent replacement or improvement made without departing from the spirit and scope of the application should be included in the protection scope of the application. In addition, the appended claims of the application are intended to cover all changes and modifications falling within the scope and boundary of the appended claims, or the equivalent form of such scope and boundary.

Claims

1. A distributed log access method based on a log framework and a network communication framework, characterized in that, The method comprises the following steps: Step S1, at the business application end, capturing log events by extending the output component of the log framework; Step S2, filtering and desensitizing the captured log events; Step S3, batch-aggregating the processed log events, and compressing the batch log data after aggregation; Step S4, encapsulating the compressed batch log data into a custom protocol format and transmitting the data to the log server through a network client that manages multiple network connections; Step S5, at the log server, performing protocol analysis and data decompression on the received data; Step S6, distributing the decompressed log data to the corresponding storage engine for storage according to the preconfigured distribution strategy.

2. The method of claim 1, wherein, In step S3, the trigger condition for batch aggregation is reaching a preset log quantity threshold or a preset time window threshold.

3. The method according to claim 1 or 2, characterized in that, In step S3, the batch log data is compressed using the Snappy algorithm.

4. The method of claim 1, wherein, In step S4, the network client manages multiple network connections through a connection pool and selects a target server node for data transmission using a weighted round-robin strategy.

5. The method of claim 1, wherein, In step S4, the custom protocol format is a binary protocol, and its encapsulation content includes a protocol header, a compressed log body, and a checksum; the protocol header contains log quantity, pre-compression size, and post-compression size information.

6. The method of claim 5, wherein, In step S5, the protocol analysis process includes length prefix-based frame decoding to solve the TCP sticking problem.

7. The method of claim 1, wherein, After step S4, step S4a is further included: if log data transmission fails or times out, retry; if it still fails after retry, write the log data to a local cache file.

8. The method of claim 7, wherein, The local cache file is stored using an in-memory mapping file.

9. The method of claim 1, wherein, In step S4, the network client and the log server maintain connection activity through a heartbeat detection mechanism.

10. The method of claim 1, wherein, The method further comprises configuration and monitoring through a management console, the configuration including setting at least one of log filtering rules, desensitization rules, batch size, compression switch, and storage routing strategy; the monitoring including monitoring at least one of log traffic, transmission delay, and error rate.

Images