Log data processing methods and systems

By introducing a unified log collection entry point and memory cache into the log storage system, generating unique identifiers and using offsets to transfer log data, the problems of duplicate log reading and copying are solved, improving the system's processing efficiency and resource utilization.

CN123115077APending Publication Date: 2026-11-17FUJIAN TIANYI WEBSOFT TECH LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611066982.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-17
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing log storage systems suffer from repeated full log readings and multiple copies when processing log data, resulting in wasted CPU, memory, and network bandwidth resources and reduced overall system efficiency, especially in high-concurrency scenarios.

Method used

A unified log collection entry service is adopted to receive log data and write it to the memory cache, generating a unique identifier. Data processing nodes pass log data through the unique identifier and offset, avoiding repeated transmission and copying, and reading the complete content from memory only when needed.

Benefits of technology

By centrally managing log data and using unique identifiers and offsets for transmission, the usage of CPU, memory, and bandwidth is significantly reduced, thereby improving the overall processing performance and resource utilization efficiency of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115077A_ABST
    Figure CN123115077A_ABST
Patent Text Reader

Abstract

This application provides a log data processing method and system, comprising: a server receiving log data reported by various business modules, writing the log data into a memory cache, and generating a unique identifier for each log data entry; sending the target unique identifier corresponding to the target log data in the memory cache to the first data processing node in the data processing link; the first data processing node reading and processing the target log data according to the target unique identifier, and passing the target unique identifier to the next data processing node in the data processing link; the server determining whether it has received a processing link completion notification sent by the last data processing node in the data processing link, and if so, releasing the storage resources corresponding to the target log data in the memory cache. This avoids repeated transmission and copying of log content between multiple data processing nodes, significantly improving the overall system processing performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of Internet data processing technology, and in particular to a log data processing method and system. Background Technology

[0002] To address different log data processing scenarios, existing log storage systems often need to store both full logs and partial log content simultaneously, and support multi-dimensional parsing and statistics.

[0003] While multi-module storage for full logs is simple to design and has low development costs, its complex processing flow can easily lead to the same log being read repeatedly during its lifecycle. For example, it might be parsed repeatedly or copied and transferred multiple times between modules. Both repeated I / O operations and data copying significantly consume CPU, memory, and network bandwidth resources, reducing overall system efficiency. Summary of the Invention

[0004] The technical problem to be solved by this invention is: how to improve the system's efficiency in processing log data.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A log data processing method, comprising: The server receives log data reported by each business module, writes the log data into a memory cache, and generates a unique identifier for each log data; it then sends the target unique identifier corresponding to the target log data in the memory cache to the first data processing node in the data processing link. The first data processing node reads and processes the target log data based on the target unique identifier, and transmits the target unique identifier to the next data processing node in the data processing link; The server determines whether it has received a processing link completion notification from the last data processing node in the data processing link. If so, it releases the storage resources in the memory cache corresponding to the target log data.

[0006] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: A log data processing system includes a server and data processing nodes. The server includes a first memory, a first processor, and a first computer program stored in the first memory and executable on the processor. When the first processor executes the first computer program, it implements the steps performed by the server as described in the log data processing method above. The data processing node includes a second memory, a second processor, and a second computer program stored in the second memory and executable on the processor; when the second processor executes the second computer program, it implements the steps performed by the data processing node in the log data processing method described above.

[0007] The beneficial effects of this invention are as follows: by receiving log data reported by various business modules from the server, centralized management and unified processing of global logs can be ensured; and after obtaining log data, the log data is written to the memory cache and a unique identifier is generated, laying the foundation for subsequent log references; enabling data processing nodes to obtain complete log data content from the memory cache through the unique identifier, and transmitting the unique identifier of the target log data between data processing nodes without transmitting the log text, thereby avoiding repeated transmission and copying of log content between multiple data processing nodes, significantly reducing the usage of CPU, memory and bandwidth, and thus significantly improving the overall processing performance of the system. Attached Figure Description

[0008] Figure 1 This is a flowchart of a log data processing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a log data processing system according to an embodiment of the present invention. Detailed Implementation

[0009] Definitions:

[0010] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0011] In existing technologies, current log storage systems often need to store both full and partial log content simultaneously, and support multi-dimensional parsing and statistics. This complex processing flow leads to the same log being read repeatedly throughout its lifecycle, possibly being parsed repeatedly or copied and transferred multiple times between modules. Both repeated I / O operations and data copying significantly consume CPU, memory, and network bandwidth resources, reducing overall system efficiency. This resource waste is particularly pronounced in high-concurrency scenarios with massive log volumes.

[0012] To address at least some of the aforementioned issues, this paper proposes an optimized architecture based on unified storage of the original log content and offset referencing. This scheme retains only one complete copy of the original log data, and all subsequent processing modules only record the offset of this log in memory. When the actual content is needed, the text fragment is read directly from memory using the offset. This approach avoids repeated transmission and copying of log content between multiple modules, significantly reducing CPU, memory, and bandwidth usage, thereby substantially improving the overall system processing performance.

[0013] The following describes a log data processing method of the present invention in detail. Please refer to [link / reference]. Figure 1 The method 100 includes steps 101 to 103: Step 101: The server receives the log data reported by each business module, writes the log data into the memory cache, and generates a unique identifier for each log data; the server sends the target unique identifier corresponding to the target log data in the memory cache to the first data processing node in the data processing link.

[0014] For example, a unified log collection entry service can be built, through which all logs generated by business modules are reported. This entry service acts as the sole hub for log processing, responsible for receiving, caching, and distributing log data, ensuring centralized management and unified processing of global logs. Upon receiving a log, the entry service immediately writes the complete log content to a local memory cache and generates a globally unique identifier (such as a UUID) as the query key. This memory cache employs a high-performance architecture, supporting fast read and write operations, laying the foundation for subsequent log retrieval.

[0015] Step 102: The first data processing node reads and processes the target log data based on the target unique identifier, and then passes the target unique identifier to the next data processing node in the data processing link. That is, when the first data processing node needs to read the original log text, it retrieves the complete content from the memory cache using the globally unique identifier ID and processes it; subsequently, it passes the globally unique identifier ID to the next data processing node in the data processing link.

[0016] Step 103: The server determines whether it has received a processing link completion notification sent by the last data processing node in the data processing link. If so, it releases the storage resources in the memory cache corresponding to the target log data.

[0017] For example, the data processing chain includes a parsing node, a statistics node, and a storage node in sequence. The parsing node is the first data processing node, and the storage node is the last data processing node. When the storage node completes the processing of the target log data, it will send a processing chain completion notification to the server. When the server receives the processing chain completion notification, it will clear the corresponding target log data in the memory cache according to the globally unique identifier ID, thereby releasing memory resources.

[0018] As described above, the beneficial effects of this invention are as follows: by receiving log data reported by various business modules from the server, centralized management and unified processing of global logs can be ensured; and after obtaining log data, the log data is written into the memory cache and a unique identifier is generated, laying the foundation for subsequent log references; enabling data processing nodes to obtain complete log data content from the memory cache through the unique identifier, and transmitting the unique identifier of the target log data between data processing nodes without transmitting the log text, thereby avoiding repeated transmission and copying of log content between multiple data processing nodes, significantly reducing the usage of CPU, memory and bandwidth, and thus significantly improving the overall processing performance of the system.

[0019] In one embodiment of the present invention, step 101, which involves writing the log data into a memory cache and generating a unique identifier for each log data entry, includes: the server constructing a key-value pair using the unique identifier as the key and the complete original content of the log data as the value, and storing the key-value pair in the memory cache.

[0020] For example, if the server receives a raw log message as follows: text: [2025-01-15 10:30:22] [INFO][order-service] User ID=12345 successfully placed an order, order number=ORD-98765, amount=299.00 yuan, product=mobile phone; and generates a unique identifier ID as: abc-123-def; then the resulting key-value pair is: key="abc-123-def", value="[2025-01-15 10:30:22][INFO] [order-service] User ID=12345 successfully placed an order, order number=ORD-98765, amount=299.00 yuan, product=mobile phone".

[0021] As described above, using a key-value pair structure to store log data enables the server to quickly locate and read the complete content of the corresponding log using a unique identifier, greatly improving data reading efficiency.

[0022] In one embodiment of the present invention, the first data processing node reading and processing the target log data according to the target unique identifier in step 102 includes: the first data processing node reading the data structure of the target log data according to the target unique identifier, and reading the target data corresponding to the target position in the target log data according to the target position obtained from the data structure.

[0023] For example, if the original log text is 1000 characters, but a certain processing step only needs 200-300 characters, since the log is a structured data format, it is easy to know what part of the data is, and the data can be retrieved according to the established rules, thereby obtaining the 200-300 characters.

[0024] As described above, by obtaining the data structure information of the log, the data processing node can accurately locate and read only the target data it needs, rather than the entire content of the log; this can significantly reduce the amount of data processing within the node and reduce local CPU and memory overhead.

[0025] In one embodiment of the present invention, step 102, in which the first data processing node reads and processes the target log data according to the target unique identifier, and transmits the target unique identifier to the next data processing node in the data processing link, includes: The first data processing node, based on the offset of the target field in the target log data obtained from the data structure, transmits the target unique identifier and the offset to the next data processing node in the data processing link; the next data processing node reads the target log data based on the target unique identifier and obtains the data in the target log data at the offset position.

[0026] As described above, by passing the target unique identifier and offset from the previous data processing node to the next data processing node, the downstream node does not need to re-parse the log data structure to determine the position of the target field. It can directly reuse the offset calculated by the upstream node. At the same time, only the "unique identifier + offset" is passed between nodes instead of the complete log text, which avoids the overhead of repeated structure parsing and further reduces CPU consumption.

[0027] In one embodiment of the present invention, the offset includes the start position and end position of the target field in the target log data; wherein, in step 102, the next data processing node reads the target log data according to the target unique identifier and obtains the data in the target log data at the offset position, which includes: the next data processing node obtains data from the target log data according to the start position and end position.

[0028] As described above, by explicitly defining the offset as "start position + end position", the data processing node can more accurately extract log segments.

[0029] In one embodiment of the present invention, step 103, in which the server determines whether it has received a processing link completion notification sent by the last data processing node in the data processing link, and if so, releases the storage resources in the memory cache corresponding to the target log data, including: the server determines whether it has received a processing link completion notification for the target log data through an asynchronous recycling program; if so, it deletes the target log data in the memory cache corresponding to the target unique identifier.

[0030] As described above, by introducing an independent asynchronous recycling program, the memory release operation is completely decoupled from the main business processing link. When the last data processing node sends a completion notification, it can end its own processing flow without synchronously waiting for the memory deletion operation to complete, thus avoiding additional delays to the business link caused by the recycling action.

[0031] In one embodiment of the present invention, step 101, which involves sending the target unique identifier corresponding to the target log data in the memory cache to the first data processing node of the data processing link, includes the server sending the target unique identifier to at least two parallel data processing links. For example, if the server needs to forward logs to downstream processing modules, such as risk control modules, statistics modules, and archiving modules, then the target unique identifier is sent to the data processing links corresponding to the risk control module, statistics module, and archiving module, respectively. Simultaneously, when the server sends the target unique identifier to multiple parallel data processing links, it needs to wait for all data processing links to report completion notifications before clearing the memory cache.

[0032] As described above, by sending the target's unique identifier to multiple parallel data processing links, the same log data can be simultaneously sent to multiple independent data processing links such as risk control, statistics, and archiving links for parallel processing. These links do not interfere with each other, greatly improving the overall efficiency of log utilization and the real-time performance of processing. Furthermore, each parallel link shares the same original log text in its memory cache, accessing it only through the unique identifier. There is no need to cache a separate copy of the log for each link, fundamentally avoiding memory resource contention and data consistency issues between multiple links. When a new log processing scenario needs to be added (such as adding a real-time recommendation link), simply send the unique identifier to the new link entry point; no modifications to existing links are required, demonstrating strong scalability.

[0033] In one embodiment of the present invention, the step 101 in which the server receives the log data reported by each business module includes: the server obtains the data information of the log request to each component through the link tracing component, and collects the data information through the ELK component to obtain the log data.

[0034] As described above, by setting up the tracing component, the call chain data generated when a request passes through each microservice component can be automatically captured. Combined with the log collection and aggregation capabilities of the ELK components (Elasticsearch + Logstash + Kibana), the log collection process is fully automated.

[0035] In one embodiment of the present invention, after the server receives the log data reported by each business module in step 101, the method further includes: the server distributing the log data to different data sources for storage.

[0036] As described above, by writing log data to a memory cache and distributing logs to different data sources, "one log, multiple persistence" is achieved, which fully meets diverse needs such as auditing, retrieval, analysis, and archiving.

[0037] This invention provides a specific application scenario to illustrate the above steps: Scenario: On an e-commerce platform, a user completes an order. The backend business module generates a raw log: text: [2025-01-15 10:30:22] [INFO][order-service] User ID=12345 Order successful, order number=ORD-98765, amount=299.00 yuan, product=mobile phone.

[0038] The system needs to perform the following operations on this log: ① collect and store; ② parse out the order number and amount for real-time risk control; ③ count the number of orders per second for monitoring; ④ archive to a disk file for auditing. In the traditional approach, each module copies or transmits the complete log string, resulting in wasted resources.

[0039] The present invention is optimized using the following method: S1, Unified Log Collection Entry Point Action: Build a log collection entry service (such as based on Kafka or a self-developed Gateway), and all business modules (order service, payment service, etc.) will report logs to this entry.

[0040] In this example: After the order service generates the above logs, it sends them to the log entry service http: / / log-collector / ingest via HTTP or RPC. This is the part of step 101 where the server receives the log data reported by each business module.

[0041] S2. Store the complete log in a memory cache and generate a unique ID. Action: The entry service receives the log, writes the original content to a local memory cache (such as Redis or ConcurrentHashMap), and generates a globally unique ID.

[0042] In this example: the original content is written to the local memory cache, and a UUID of abc-123-def is generated. Then, the memory cache stores the key-value pair: key="abc-123-def", value="[2025-01-15 10:30:22] [INFO][order-service] User ID=12345 Order successful, order number=ORD-98765, amount=299.00 yuan, product=mobile phone". This corresponds to the log data writing part of step 101.

[0043] S3, Only a unique ID is passed between subsequent modules. Action: When it is necessary to forward logs to downstream processing modules, only the unique ID is passed instead of the entire log text.

[0044] In this example, the log entry service pushes the UUID: abc-123-def to the message queues of three downstream modules: Risk control queue: Message body {"logId":"abc-123-def"}; Statistics queue: Message body {"logId":"abc-123-def"}; Archive queue: Message body {"logId":"abc-123-def"}; The network transmission volume decreased from approximately 150 bytes (the entire log) to approximately 40 bytes.

[0045] S4. When extracting partial content, pass the ID + offset. Action: If a module only needs some fields from the log, it passes "log ID + start position + end position", and the receiver extracts the substring from memory based on the offset.

[0046] In this example: The risk control module needs to check the order number and amount. The module knows the offset of the target fields in the log; for example, "Order Number=" starts at byte 65 with a length of 9 bytes, and "Amount=" starts at byte 80 with a length of 6 bytes. Therefore, the risk control module sends the following message to the downstream "risk control rule engine": {"logId":"abc-123-def","extractions":[{"start":65, "end":74},{"start":80,"end":86}]}. Upon receiving the message, the risk control rule engine reads the original log from its memory cache using the UUID, then extracts "ORD-98765" and "299.00" based on the offset, and performs risk control checks, such as whether the amount exceeds the limit. The entire original log text is not transmitted during this process.

[0047] S5. Asynchronously reclaim memory after the processing chain is completed. Action: Once the log has been processed by all necessary modules (e.g., parsing, statistics, and storage are all complete), the last module sends a "completion notification" to the asynchronous recycling program, which then deletes the log entry from the memory cache.

[0048] In this example: After the archiving module persists the logs to disk, it determines that this is the last link in the processing chain, so it sends the message {"logId":"abc-123-def","action":"complete"} to the recycling service. Upon receiving this message, the recycling service deletes the key abc-123-def from the memory cache, freeing up space.

[0049] The three link processing procedures include: Order Service → Log Entry (stored in memory, ID generated); ├─(Pass ID)→Risk Control Module→(ID + Offset)→Rule Engine→Completion Notification; ├─(Pass ID)→Statistics Module→(ID + Offset)→Aggregation Calculation→Completion Notification; └─(Pass ID)→Archiving Module→(Read Full Text and Write to Disk)→Completion Notification (Finally triggers recycling).

[0050] Overall Results: Throughout the process, the original log content was stored only once in the memory of the entry service; the risk control, statistics, and archiving modules only exchanged lightweight IDs (or ID + offset), without repeated transmission or copying of log text; CPU usage was significantly reduced due to the avoidance of multiple string concatenations / copying, resulting in a reduction in memory consumption and saving over 70% of network bandwidth; the system's throughput under high concurrency scenarios was greatly improved. These examples demonstrate that the core idea of ​​the invention is to use references (ID / offset) instead of the data itself for transmission, reading only when necessary from a single storage location, thereby greatly reducing resource consumption.

[0051] The present invention also provides a log data processing system, including a server and a data processing node. The server includes a first memory, a first processor, and a first computer program stored in the first memory and executable on the processor. When the first processor executes the first computer program, it implements the steps performed by the server as described in the log data processing method above. The data processing node includes a second memory, a second processor, and a second computer program stored in the second memory and executable on the processor. When the second processor executes the second computer program, it implements the steps performed by the data processing node as described in the log data processing method above.

[0052] Please refer to Figure 2 In a log data processing system, the server and data processing nodes can also be implemented based on the same memory and processor, and can be different modules in the same computer program.

[0053] In summary, the log data processing method and system provided in this application, based on an optimized architecture of unified storage of original log content and offset referencing, retains only one complete copy of the original log data. When all subsequent processing modules process the log data, they only record the offset of the log in memory (such as the start and end positions). When the actual content is needed, the text fragment is read directly from memory using the offset. This approach completely avoids the repeated transmission and copying of log content between multiple modules, and each link executes in parallel without interference, fully releasing the parallel computing capabilities of the distributed system, significantly increasing the number of log entries processed per unit time, and greatly reducing the usage of CPU, memory, and bandwidth, thereby significantly improving the overall processing performance of the system.

[0054] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A log data processing method, characterized in that, include: The server receives log data reported by each business module, writes the log data into the memory cache, and generates a unique identifier for each log data. Send the unique identifier of the target log data in the memory cache to the first data processing node in the data processing link; The first data processing node reads and processes the target log data based on the target unique identifier, and transmits the target unique identifier to the next data processing node in the data processing link; The server determines whether it has received a processing link completion notification from the last data processing node in the data processing link. If so, it releases the storage resources in the memory cache corresponding to the target log data.

2. The log data processing method according to claim 1, characterized in that, The step of writing the log data to a memory cache and generating a unique identifier for each log data entry includes: The server constructs a key-value pair using the unique identifier as the key and the complete original content of the log data as the value, and stores the key-value pair in the memory cache.

3. The log data processing method according to claim 1, characterized in that, The first data processing node reads and processes the target log data based on the target's unique identifier, including: The first data processing node reads the data structure of the target log data according to the target unique identifier, and reads the target data corresponding to the target location in the target log data according to the target location obtained from the data structure.

4. The log data processing method according to claim 3, characterized in that, The first data processing node reads and processes the target log data based on the target unique identifier, and passes the target unique identifier to the next data processing node in the data processing link, including: The first data processing node, based on the offset of the target field in the target log data obtained from the data structure, transmits the target unique identifier and the offset to the next data processing node in the data processing link; The next data processing node reads the target log data based on the target unique identifier and obtains the data at the offset position in the target log data.

5. The log data processing method according to claim 4, characterized in that, The offset includes the start and end positions of the target field in the target log data; The next data processing node reads the target log data based on the target unique identifier and obtains the data at the offset position in the target log data, including: The next data processing node retrieves data from the target log data based on the start and end positions.

6. The log data processing method according to claim 1, characterized in that, The server determines whether it has received a processing completion notification from the last data processing node in the data processing chain. If so, it releases the storage resources in the memory cache corresponding to the target log data, including: The server determines whether it has received a completion notification of the processing link for the target log data through an asynchronous recycling program. If it does, it deletes the target log data corresponding to the target's unique identifier from the memory cache.

7. The log data processing method according to claim 1, characterized in that, The step of sending the target unique identifier corresponding to the target log data in the memory cache to the first data processing node in the data processing link includes: The server sends the target unique identifier to at least two parallel data processing links.

8. The log data processing method according to claim 1, characterized in that, The server receives log data reported by each business module, including: The server obtains data information from each component in the log request through the link tracing component, and collects the data information through the ELK component to obtain the log data.

9. A log data processing method according to claim 1, characterized in that, After receiving the log data reported by each business module, the server also includes: The server distributes the log data to different data sources for storage.

10. A log data processing system, characterized in that, The method includes a server and a data processing node. The server includes a first memory, a first processor, and a first computer program stored in the first memory and executable on the processor. When the first processor executes the first computer program, it implements the steps performed by the server in the log data processing method according to any one of claims 1 to 9. The data processing node includes a second memory, a second processor, and a second computer program stored in the second memory and executable on the processor; when the second processor executes the second computer program, it implements the steps performed by the data processing node in the log data processing method according to any one of claims 1 to 9.