A distributed log management method, system, computer equipment, and storage medium based on global logical offsets.
By using a global logical offset mechanism, the progress mapping problem caused by log file changes in the distributed log management system is solved, which ensures the accuracy of data collection progress and the reliability of anomaly recovery, thus guaranteeing the integrity and consistency of log data.
Patent Information
- Application Number
- CN202611239886.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-17
- Publication Date
- 2026-11-17
AI Technical Summary
In distributed log management systems, existing technologies cannot effectively map progress when log files are rotated, renamed, truncated, or deleted, leading to state deviations and blind rollbacks in anomaly recovery, making it difficult to guarantee the determinism and reliability of recovery.
A global logical offset mechanism is adopted. Through the collaborative work of the log collection end and the management server, the log collection end determines the current physical reading position and segment ledger, calculates the global logical offset, and the log management server performs continuity verification to ensure the accuracy and reliability of the collection progress.
It enables accurate mapping of collection progress when log files change, avoids collection interruptions caused by invalid file identifiers, improves the certainty and reliability of anomaly recovery, and ensures the integrity and consistency of log data.
Smart Images

Figure CN123114883A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of log collection and operation and maintenance monitoring technology, and in particular to a distributed log management method, system, computer equipment and storage medium based on global logical offset. Background Technology
[0002] With the widespread adoption of distributed service architectures, business systems are typically deployed across hundreds or thousands of nodes. This results in a surge in application logs, system logs, audit logs, and host status logs. Therefore, the continuous and reliable collection of this log data and its unified transmission to a centralized storage platform for subsequent retrieval and analysis has become a critical engineering requirement for operations monitoring, troubleshooting, and security auditing. However, the inherent characteristics of distributed environments—multi-node deployment, complex network communication, dynamic scaling of service instances, and frequent log file rotation—pose technical challenges to log management systems in terms of data integrity and state consistency. Ensuring that every log entry is accurately captured and not lost in large-scale, high-concurrency collection scenarios, and achieving rapid self-healing after various anomalies, has become a core issue of ongoing concern in this field.
[0003] In related technologies, the implementation of the collection program in log management can be summarized as identifying the target file and recording and tracking its read position. Specifically, the collection program first uniquely determines which physical file is being read by combining several file attributes. Then, it records the byte offset or line number read within that file in local persistent storage. For example, it saves the file identifier and corresponding current read position in a local file, or identifies the file based on its inode and device number and records the read progress in a status file. Furthermore, log management also involves periodically scanning the target directory and adjusting the starting position according to the read progress; using a combination of real-time and non-real-time offset recording files to achieve breakpoint resumption; uploading the collected logs to a message queue and generating corresponding metadata for analysis and processing; and implementing reliability assurance schemes for log collection, transmission, and retrieval scenarios.
[0004] In the process of implementing the relevant technologies, it was recognized that the technologies at least have the following technical problems: When log files are rotated, renamed, truncated, deleted, or archived and migrated, the file identity on which the original state records depend becomes invalid or changes, making it impossible to map the existing progress to the currently existing physical files. At the same time, in the mode that relies solely on the client's local offset file to advance the collection status, when situations occur such as log data being read but not yet confirmed by the server, or the server's memory progress advancing but failing to be persisted in time after a crash, discrepancies can easily arise between the state recorded by the client and the state actually received by the server. This makes it impossible to perceive and correct the state discrepancy in a timely manner, and the abnormal recovery process can only rely on the client's local state to blindly roll back, making it difficult to effectively guarantee the determinism and reliability of the recovery. Summary of the Invention
[0005] In view of this, this application provides a distributed log management method, system, computer device and storage medium based on global logical offsets. The main purpose is to solve the problems that it is currently impossible to map existing progress to the currently existing physical files, the abnormal recovery process can only rely on the local state of the client for blind rollback, and the determinism and reliability of recovery are difficult to guarantee effectively.
[0006] According to a first aspect of this application, a distributed log management method based on global logical offsets is provided. This method is applied to a log management system, which includes a log collection terminal and a log management server. The method includes: The log collection terminal determines the currently managed log stream, locates the target log file of the current application in the log stream, and the log stream corresponds to a segment ledger for recording multiple logical segments connected end to end in chronological order. Each logical segment corresponds to a log file generated during the log rotation process. The log collection terminal obtains the current physical read position of the target log file, and determines the global logical offset by combining the current physical read position with the segmented ledger. The current physical read position is used to locate the actual starting point of the log data to be read in the target log file, and the global logical offset is used to identify the continuous progress of the current physical read position in the log stream. The log collection terminal reads new log data from the target log file according to the actual starting point located by the current physical reading position. Based on the local offset of the new log data in the target log file and the global logical offset, it calculates the logical start offset and logical end offset, encapsulates the new log data, the logical start offset, and the logical end offset into a log batch, and sends the log batch to the log management server. The log management server determines the expected next offset for the log stream maintenance, and performs continuity verification on the log batch by comparing the logical starting offset with the expected next offset. Upon receiving a successful verification response from the log management server, the log collection terminal extracts the confirmation offset carried in the successful verification response, and updates and saves the current physical read position and the global logical offset based on the confirmation offset to complete the management of the log stream.
[0007] According to a second aspect of this application, a distributed log management system based on global logical offsets is provided, the log management system including a log collection terminal and a log management server: The log collection terminal is used to determine the currently managed log stream, locate the target log file of the current application in the log stream, and the log stream corresponds to a segment ledger for recording multiple logical segments connected end to end in chronological order. Each logical segment corresponds to a log file generated during the log rotation process. The log collection terminal is also used to obtain the current physical read position of the target log file, and determine the global logical offset by combining the current physical read position and the segmented ledger. The current physical read position is used to locate the actual starting point of the log data to be read in the target log file, and the global logical offset is used to identify the continuous progress of the current physical read position in the log stream. The log collection terminal is also used to read new log data from the target log file according to the actual starting point located by the current physical reading position, calculate the logical start offset and logical end offset according to the local offset of the new log data in the target log file and the global logical offset, encapsulate the new log data, the logical start offset and the logical end offset into a log batch, and send the log batch to the log management server. The log management server is used to determine the expected next offset to be maintained for the log stream, and to perform continuity verification on the log batch by comparing the logical starting offset with the expected next offset. The log collection terminal is also used to extract the confirmation offset carried in the verification success response when it receives the verification success response returned by the log management server, and update and save the current physical read position and the global logical offset according to the confirmation offset, so as to complete the management of the log stream.
[0008] According to a third aspect of this application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the first aspects above.
[0009] According to a fourth aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.
[0010] By employing the above technical solutions, this application provides a distributed log management method, system, computer device, and storage medium based on global logical offsets. The log collection end in this application maintains a segment ledger for each log stream, mapping multiple log files generated during rotation to logically connected segments on a logical address axis. This eliminates the dependence of collection progress on specific log file identities. Therefore, when log files are rotated, renamed, or truncated, the existing progress can still be mapped to the currently existing physical log file through the segment ledger, avoiding collection interruptions caused by invalid file identities. Simultaneously, the log management server independently maintains the expected next offset and performs continuity checks. The log collection end only advances and persists the local checkpoint after receiving the confirmed offset from the log management server, forming a closed-loop feedback mechanism with the log management server as the authoritative benchmark. Thus, when situations occur where the log management server's memory progress has advanced but not yet been persisted, the log collection end can still recover from the last confirmed position without blindly rolling back, improving the certainty and reliability of abnormal recovery.
[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description
[0012] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This illustration shows a flowchart of a distributed log management method based on global logical offsets provided in an embodiment of this application. Figure 2 This illustration shows a timing diagram of a log data collection process provided in an embodiment of this application; Figure 3 This illustration shows a timing diagram of an abnormality handling process provided in an embodiment of this application; Figure 4 A timing diagram illustrating a network problem and retry process provided in an embodiment of this application is shown. Figure 5 This illustration shows a logical architecture diagram of a distributed log management method based on global logical offsets provided in an embodiment of this application; Figure 6 This illustration shows a schematic diagram of a distributed log management system based on global logical offsets provided in an embodiment of this application. Figure 7 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation
[0013] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.
[0014] This application provides a distributed log management method based on global logical offsets, such as... Figure 1 As shown, this method is applied to a log management system, which includes a log collection terminal and a log management server. The method includes: S10: The log collection end determines the currently managed log stream and locates the target log file of the current application within the log stream.
[0015] In this embodiment, when the log collection terminal executes a log collection task, it first determines the log stream that needs to be managed. A log stream is a logical unit divided according to the log source, used to uniquely identify a series of log data from the same collection agent, the same node, the same log path, and the same log type. After starting the collection task, the log collection terminal determines the currently managed log stream based on the configured log path and collection rules, and locates the target log file currently in an active writing state in the storage directory corresponding to the log stream. Further, the log stream corresponds to a segment ledger, which records multiple logical segments connected end-to-end in chronological order. Each logical segment corresponds to a physical log file generated during log rotation and records the logical start address and logical end address of that logical segment. The log collection terminal determines the log stream and locates the target log file, thereby clarifying the target of the current collection task and providing a basis for subsequently obtaining the read position, reading log data, and calculating the offset.
[0016] In this way, during the above process, the log stream will logically divide the log data scattered across different nodes and directories, so that the data from the same log source can form a unified management view in a logical manner. At the same time, the segmented ledger records each physical log file generated in the log stream and its location in the logical address space in the form of logical segments, thereby providing a structured basis for subsequent incremental collection and status management.
[0017] For example, suppose an application's log path is / var / log / app / . When the logs are rotated, there may be multiple physical log files such as app.log, app.log.1, and app.log.2 under this path. The log collection end will determine the log stream corresponding to the log path and locate the app.log file that is currently being written in the log stream as the target log file.
[0018] S20: The log collection terminal obtains the current physical read position of the target log file, and determines the global logical offset by combining the current physical read position with the segment ledger.
[0019] In this embodiment, after locating the target log file, the log collection terminal obtains the current physical read position of the target log file. The current physical read position is used to locate the actual starting point of the log data to be read in the target log file. The current physical read position includes at least the path identifier of the target log file and the byte offset within the file, indicating from which specific byte position in the target log file to start reading the new log content. After obtaining the current physical read position, the log collection terminal queries the segment ledger to see if there is a logical segment corresponding to the target log file, based on the file identity information of the target log file corresponding to the current physical read position. If there is a corresponding logical segment in the segment ledger, the logical starting address of the logical segment is obtained, and the global logical offset is calculated based on the logical starting address and the current physical read position. If there is no corresponding logical segment in the segment ledger, the target log file is determined to be the first physical file to appear in the current log stream. The log collection terminal creates a corresponding new logical segment for the target log file, sets the logical starting address of the new logical segment to the current cumulative length value of the log stream, and determines the global logical offset based on the logical starting address and the current physical read position. The global logical offset is used to identify the continuous progress of the current physical read position in the log stream, that is, the unique coordinates of the read position in the entire logical address space.
[0020] In this way, by converting the byte offsets within the physical file into logical offsets in the global logical address space during the above process, the scattered physical file reading progress can be unified onto the continuous logical address axis of the log stream.
[0021] Continuing with the previous example, assuming the logical starting address of the logical segment corresponding to app.log in the segmented ledger is 1000, and the current physical read position is byte offset 200 within the app.log file, then the log collection end calculates the corresponding global logical offset to be 1200.
[0022] S30: The log collection terminal reads the new log data from the target log file according to the actual starting point located by the current physical reading position. Based on the local offset and global logical offset of the new log data in the target log file, it calculates the logical start offset and logical end offset, encapsulates the new log data, logical start offset and logical end offset into a log batch, and sends the log batch to the log management server.
[0023] In this embodiment, after obtaining the current physical read position, the log collection terminal uses the local offset corresponding to that current physical read position within the target log file as the starting local offset. Starting from the actual starting point corresponding to this starting local offset, it reads new log data from the target log file. During the reading process, the log collection terminal uses the minimum local offset corresponding to the currently read log data within the target log file as the starting local offset and the maximum local offset corresponding to the currently read log data within the target log file as the ending local offset. Based on the global logical offset and the starting local offset, it calculates the logical starting offset of the first log record in the currently read log data, and based on the global logical offset and the ending local offset, it calculates the logical ending offset of the last log record in the currently read log data. The log collection terminal encapsulates the currently read log data, the logical starting offset, and the logical ending offset into a log batch. The log batch includes at least the above three pieces of information and may further include metadata such as the collection node identifier, log type, and file identifier. Subsequently, the log collection terminal sends the log batch to the log management server, thereby binding the log data content and global logical offset information into a transmission unit, enabling the log management server to obtain its position in the global logical address space while receiving the log content.
[0024] For example, suppose the log collection terminal reads 1024 bytes of new log data from byte offset 200 in app.log, with a starting local offset of 200 and an ending local offset of 1224. Then the logical starting offset is 1200 and the logical ending offset is 2224. The log collection terminal encapsulates the read log content, the logical starting offset of 1200, and the logical ending offset of 2224 into a log batch and sends it to the log management server.
[0025] S40: The log management server determines the expected next offset for maintaining the log stream and performs continuity verification on the log batch by comparing the logical starting offset with the expected next offset.
[0026] In this embodiment, after receiving a log batch from the log collection terminal, the log management server obtains the expected next offset maintained for that log stream. The expected next offset is a state value independently maintained by the log management server for each log stream, used to indicate the logical start offset of the next log batch the log management server expects to receive, i.e., the logical end offset position of the last log data successfully received by the log management server. The log management server extracts the logical start offset from the received log batch and compares it with the expected next offset. If the logical start offset matches the expected next offset, the log batch is determined to have passed the continuity check. The log management server receives the log batch and updates the expected next offset to the value of the logical end offset to indicate the next expected logical start offset. If the logical start offset does not match the expected next offset, the log batch is determined to have failed the continuity check. The log management server records the current expected next offset to instruct the log collection terminal to roll back to the reading position corresponding to the current expected next offset and returns a check failure response to the log collection terminal, carrying the expected next offset.
[0027] In this way, by independently verifying the offset reported by the log collection terminal through the log management server, a consistency check mechanism with the log management server as the authoritative benchmark can be realized, so that the state deviation between the log collection terminal and the log management server can be detected in a timely manner during the transmission process.
[0028] Continuing with the previous example, the expected next offset maintained by the log management server for this log stream is 1200, and the logical starting offset in the log batch is 1200. Since the two are consistent, the log management server determines that the verification is successful, receives the batch, and updates the expected next offset to 2224.
[0029] S50: When the log collection terminal receives a successful verification response from the log management server, it extracts the confirmation offset carried in the successful verification response, updates and saves the current physical read position and the global logical offset based on the confirmation offset, so as to complete the management of the log stream.
[0030] In this embodiment, after sending the log batch to the log management server, the log collection terminal waits for a response from the log management server. When the log collection terminal receives a successful verification response from the log management server, it extracts the confirmation offset from the response. The confirmation offset is the offset value obtained by the log management server after passing continuity verification and updating the expected next offset according to the logical end offset in the log batch. It indicates the latest global logical offset position successfully processed by the log management server and is numerically equal to the updated expected next offset. The log collection terminal queries the segment ledger based on the confirmation offset, reverse-locates the target physical file and its position within the file corresponding to the confirmation offset, updates the current physical read position to the position within that file, updates the global logical offset to the confirmation offset, and then persistently saves the updated current physical read position and global logical offset as local checkpoints.
[0031] In this way, during the above process, the log collection end only advances the local checkpoint and persists the data after receiving a clear success confirmation from the log management server, thereby ensuring that the local state is synchronized with the log management server state and avoiding incorrectly advancing the collection progress in the event of communication abnormalities or server processing failures.
[0032] Continuing with the previous example, the log management server returns a confirmation offset of 2224. The log collection end queries the segment ledger based on this offset, locates the corresponding byte offset position of 1224 in the app.log file, updates the current physical read position to the 1224-byte offset of app.log, and updates the global logical offset to 2224 before persisting it. This completes the current round of log collection, and subsequent log collection will continue to read from this position.
[0033] In this embodiment, obtaining the current physical read position of the target log file and determining the global logical offset by combining the current physical read position and the segment ledger includes: querying the segment ledger to see if there is a target logical segment corresponding to the target log file, based on the file identity information of the target log file, where the file identity information includes at least the path information and file system identifier information of the target log file; if the target logical segment exists in the segment ledger, obtaining the logical starting address of the target logical segment and determining the global logical offset based on the sum of the logical starting address and the local offset of the current physical read position within the target log file; if the target logical segment does not exist in the segment ledger, determining that the target log file is the first physical file to appear in the log stream, creating a corresponding new logical segment for the target log file, setting the logical starting address of the new logical segment to the current cumulative length value of the log stream, and determining the global logical offset based on the sum of the logical starting address of the new logical segment and the local offset of the current physical read position within the target log file.
[0034] In this embodiment, after starting the collection task, the log collection terminal discovers log files using a hybrid mode combining file system event monitoring and periodic verification. File system event monitoring quickly detects events such as additions, modifications, closures, renamings, and deletions of files in the target directory, enabling real-time responses to log file changes. Periodic verification serves as a fallback mechanism, scanning the target directory and reconstructing the local file state after event loss, remote file system instability, or a restart of the collection terminal process, ensuring the reliability and integrity of file discovery. Once the log collection terminal locates the target log file using the above mechanism, it obtains the current physical read position of the target log file. This current physical read position is used to locate the actual read point in the local log file, and may include, for example, the path information of the target log file and the byte offset within that file, to accurately pinpoint the actual starting point of the log data to be read.
[0035] After obtaining the current physical read location, the log collection end queries the segment ledger based on the file identity information of the target log file to determine whether the target log file has been recorded in the log stream. The segment ledger is a data structure maintained independently by the log collection end for each log stream, used to record multiple logical segments in the log stream that are sequentially linked together, forming a logical segment sequence. , , Wait, each logical segment must record at least the segment number, the file identity information of the corresponding physical file, the segment status, the segment logical start address, and the segment logical end address, etc. The file identity information is not limited to the file name, but may include at least one or more of the file's absolute path, the time of first discovery, and the file size when closed, to help identify the uniqueness of the file in scenarios such as file name changes or renaming.
[0036] The log collection terminal compares the path information and file system identifier information in the current physical read location with the file identity information recorded in each logical segment of the segment ledger. If a matching target logical segment is found, the logical starting address of that target logical segment is obtained. And based on the local offset of the logical starting address and the current physical read position within the target log file. The sum of these values is used to calculate and determine the global logical offset. , that is If no matching target logical segment is found, the target log file is determined to be the first physical file to appear in the current log stream. The log collection end creates a corresponding new logical segment for the target log file, sets the logical starting address of the new logical segment to the current cumulative length value of the log stream, and calculates the global logical offset based on the sum of the logical starting address and the local offset of the current physical read position within the target log file. Simultaneously, the new logical segment is added to the segment ledger. For any log stream, when a new segment is generated due to log rotation, renaming, truncation, or new file activation... season In other words, the logical start address of the new segment is equal to the logical end address of the previous logical segment, thus ensuring the continuity of logical segments within the same log stream on the logical address axis. Based on this, the base address of the current active file is defined as the sum of the cumulative lengths of closed and confirmed historical segments and the cumulative lengths of deleted but still retained historical segments in the logical ledger. The global logical offset currently recorded in the log stream is equal to this base address plus the local offset of the current physical read position within the target log file.
[0037] In this way, through the segmented ledger mechanism described above, the local offset in the current physical read position represents the starting point of reading the log data to be read within a single physical file, while the global logical offset maps this local offset to a unique coordinate in the continuous logical address space of the entire log stream. This is used to identify the continuous position of the read progress in the entire collection chain, which is convenient for subsequent server-side verification, breakpoint resumption, and duplicate data identification. Moreover, the segmented ledger maps multiple physical files generated by rotation to a continuous logical segment sequence of the same log stream, so that the global logical offset does not depend on the identity of a specific physical file. Even if the file name of the target log file changes or its storage path changes, as long as the log collection end can locate the corresponding logical segment in the segmented ledger through the file identity information, it can accurately recover the current read position from the logical address space, which can avoid the loss of collection progress caused by file rotation or renaming.
[0038] For example, suppose a log stream has generated two historical files, app.log.1 and app.log.2, as well as the currently active app.log file due to rotation. The segment ledger records three corresponding logical segments in chronological order: The logical start address is 0 and the logical end address is 1000. The logical start address is 1000 and the logical end address is 2000. The logical start address is 2000, and the logical end address is yet to be determined. When the log collection end locates the target log file as app.log, the current physical read position is byte offset 300 within the app.log file, and the corresponding logical segment of the file is found to be... When its logical starting address is 2000, the log collection terminal calculates the global logical offset to be 2300. This offset indicates the precise logical position of the current reading progress in the log stream.
[0039] In this embodiment, optionally, new log data is read from the target log file according to the actual starting point located at the current physical read position. Based on the local offset and global logical offset of the new log data within the target log file, a logical start offset and a logical end offset are calculated. The new log data, the logical start offset, and the logical end offset are then encapsulated into a log batch. This includes: using the local offset corresponding to the current physical read position within the target log file as the read start local offset, and determining the actual starting point corresponding to the read start local offset; starting from the actual starting point, new log data is read from the target log file according to preset encapsulation trigger conditions, wherein the encapsulation... The triggering conditions include one or more of the following: the amount of log data read reaches a preset size threshold, the number of log lines read reaches a preset number of lines threshold, and the reading duration reaches a preset time window. Within the target log file, the minimum local offset corresponding to the newly added log data is determined as the starting local offset, and the maximum local offset corresponding to the newly added log data within the target log file is determined as the ending local offset. The logical starting offset is calculated based on the sum of the global logical offset and the starting local offset, and the logical ending offset is calculated based on the sum of the global logical offset and the ending local offset. The number of newly added logs, the logical starting offset, and the logical ending offset are encapsulated to obtain the log batch.
[0040] In this embodiment, the log collection terminal is equipped with a file reading module. After obtaining the current reading position, the file reading module uses the local offset corresponding to the current physical reading position within the target log file as the starting local offset for reading, determines the actual starting point corresponding to the starting local offset, and then continuously reads new log content from the target log file starting from this actual starting point. During the reading process, the file reading module divides the read log data according to preset encapsulation triggering conditions to determine the boundary of a log batch. The encapsulation triggering conditions can be implemented in one or more combinations of various forms. For example, encapsulation can be triggered according to a preset data size threshold, i.e., a batch is formed when the amount of log data read reaches a preset size threshold, such as 1MB; or encapsulation can be triggered according to a preset line number threshold, i.e., a batch is formed when the number of log lines read reaches a preset number of lines, such as 1000 lines; or encapsulation can be triggered according to a preset time window, i.e., a batch is formed when the reading duration reaches a preset time window, such as 5 seconds; in some embodiments, encapsulation can also be triggered according to log boundaries, i.e., a batch is formed when a specific log separator or structural marker appears in the log content. This application does not limit the specific content of the encapsulation triggering conditions. The encapsulation triggering conditions can be flexibly combined according to the actual business scenario. For example, a size threshold and a time window can be set at the same time, and the encapsulation can be triggered based on the first one reached. This ensures that log batches can be sent in a timely manner in the event of a sudden or sparse log traffic.
[0041] After determining the log data for a batch, the file reading module identifies the minimum local offset corresponding to the newly added log data in the target log file as the starting local offset, and the maximum local offset corresponding to the newly added log data in the batch as the ending local offset. Then, based on the sum of the acquired global logical offset and the starting local offset, the file reading module calculates the logical starting offset of the first log record in the batch (logical starting offset = global logical offset + starting local offset); and the logical ending offset of the last log record in the batch (logical ending offset = global logical offset + ending local offset). Through these calculations, the byte offset range within the physical file is mapped to a continuous offset range in the global logical address space of the log stream. Subsequently, the file reading module encapsulates the newly added log data itself, along with the calculated logical starting and ending offsets, into a log batch. This log batch may further include information such as the collection node identifier, log type, and file identifier, so that the log management server can identify the source and log stream to which the batch belongs upon receipt. After encapsulation, the file reading module writes the log batch into a blocking queue, waiting for the log sending module set in the log collection terminal to retrieve the batch from the blocking queue and send it to the log management server. The blocking queue is used to decouple the log reading process from the log sending process, so that the file reading module does not need to directly depend on the server response speed. As long as the queue is not full, the file reading module can continuously read local logs, while the sending module set in the log collection terminal retrieves the batch to be sent from the queue according to its own processing capacity and sends it to the log management server.
[0042] In this way, by using the above-mentioned method of batch encapsulating log data based on encapsulation trigger conditions, the log collection end can balance collection efficiency and transmission real-time performance. When the log traffic is large, larger batches can be formed by triggering with size thresholds or number of lines to improve throughput. When the log traffic is small, time windows can be used to ensure that logs can be sent in a timely manner to reduce latency. At the same time, by mapping the local offset range in the physical file to the global logical offset range, each log batch has a definite start and end position in the logical address space. This provides a precise location basis for the log management server to perform offset continuity verification in the future. It also enables the accurate location of unconfirmed offset positions for continued transmission when an anomaly occurs and recovery is needed, avoiding data duplication or omission.
[0043] For example, suppose the file reading module in the log collection terminal obtains the current physical read position as byte offset 300 in the app.log file, with a corresponding global logical offset of 2300. The encapsulation trigger condition is set to trigger encapsulation once every 1000 lines of logs read. The file reading module starts reading from byte offset 300 in the app.log file. When 1000 lines have been read, the minimum local offset of these 1000 log lines in the file is 300, and the maximum local offset is 2500. Based on the global logical offset of 2300, the logical start offset of this batch is calculated to be 2600, and the logical end offset is 4800. The file reading module encapsulates the 1000 log lines, the logical start offset of 2600, the logical end offset of 4800, as well as the collection node identifier, log type, and file identifier into a log batch, writes it to a blocking queue, and waits to send it to the log management server.
[0044] In this embodiment of the application, optionally, the expected next offset maintained for the log stream is determined, and the continuity of the log batch is verified by comparing the logical start offset with the expected next offset. This includes: obtaining the expected next offset maintained for the log stream, whereby the expected next offset indicates the logical start offset of the expected next log batch; extracting the logical start offset from the log batch, comparing whether the logical start offset is consistent with the expected next offset to determine whether the log batch has successfully passed the continuity verification; correspondingly, the method further includes: if the logical start offset is consistent with the expected next offset, then the log batch is determined to have successfully passed the continuity verification, the log management server updates the expected next offset according to the logical end offset, and uses the updated expected next offset as the confirmation offset, generates a verification success response carrying the confirmation offset, and returns the verification success response to the log collection end; if the logical start offset is inconsistent with the expected next offset, then the log batch is determined to have failed the continuity verification, the log management server generates a verification failure response carrying the expected next offset, and returns the verification failure response to the log collection end.
[0045] In this embodiment, after the log sending module in the log collection terminal obtains the log batch to be sent from the blocking queue, it sends the log batch to the log management server. Upon receiving the log batch, the log management server first obtains the expected next offset maintained for this log stream. This expected next offset indicates the logical start offset of the next log batch that the log management server expects to receive, i.e., the logical end offset position of the last log data successfully received by the log management server, representing the latest collection progress currently recognized by the log management server. Subsequently, the log management server extracts the logical start offset from the received log batch and compares it with the expected next offset to determine whether the log batch has successfully passed the continuity check. During the verification process, the log management server simultaneously verifies the log format, node information, and log type in the batch to ensure the integrity and compliance of the batch data. If the logical starting offset matches the expected next offset, the log management server determines that the log batch has successfully passed the continuity verification, indicating that the log data in the batch is logically connected to the received log data without any jumps or overlaps. Subsequently, the log management server updates the expected next offset to the value of the logical ending offset in the log batch and uses the updated expected next offset as the confirmation offset. It then generates a verification success response carrying the confirmation offset and returns the verification success response to the log collection end to notify the log collection end that the batch has been successfully received. The confirmation offset represents the latest global logical offset position that the log management server has successfully processed. If the logical starting offset is inconsistent with the expected next offset, the log management server determines that the log batch has failed the continuity check. At this time, the log management server will not accept the batch and will generate a check failure response carrying the current expected next offset. The server will return the check failure response to the log collection end to notify the log collection end that there is a deviation between the offset currently reported and the offset expected by the server and that it needs to be corrected.
[0046] Thus, through the above process, even if the log collection end process restarts, the machine crashes, or the network is abnormal, because the local checkpoint always remains at the position last confirmed by the log management server, the log collection end can continue to read and send log data from that confirmed position after recovery, avoiding log loss due to premature offset submission. Moreover, through the log management server's multiple checks on batch offset, format, node information, log type, and continuity, as well as the closed-loop mechanism of submitting offsets only after confirmation by the log management server, reliable transmission of logs from local files to the server can be achieved, ensuring the consistency between the local state of the log collection end and the state of the log management server, providing basic reliability and recoverability for the collection link.
[0047] For example, suppose the log management server maintains an expected next offset of 4800 for this log stream. The log sending module in the log collection terminal reports a log batch with a logical start offset of 4800 and a logical end offset of 6300. After the log management server compares and confirms that the verification is successful, it updates the expected next offset to 6300 according to the logical end offset and sends 6300 as the confirmation offset in the success response back to the log collection terminal. Upon receiving this, the log collection terminal updates its local checkpoint to the global logical offset of 6300 and persists it. If the logical start offset in the reported batch is 4500, which is inconsistent with the expected next offset of 4800, the log management server determines that the verification has failed and returns a failure response carrying the expected next offset of 4800, notifying the log collection terminal that it needs to be corrected and restored from the logical offset of 4800.
[0048] In summary, the timing diagram of the log data collection process described in the above embodiments is as follows: Figure 2 As shown, the interactive entities include the file reading module of the log collection terminal, the offset management module, the blocking queue, the log sending module set in the log collection terminal, and the log management server. First, the file reading module sends a request to the offset management module to obtain the current read position and the global logical offset. After receiving this request, the offset management module returns the current file position and its corresponding offset to the file reading module. After obtaining the current read position, the file reading module continuously reads newly added log data from the local log file according to preset encapsulation conditions, and then combines the read log data with the calculated logical start. Logical End Offset The logs are packaged together into a LogBatch, which is then written to the send queue, also known as the blocking queue. The log sending module retrieves the LogBatch to be sent from the blocking queue and sends it to the log management server. After receiving the log batch, the log management server verifies the offset continuity and other information. If the verification passes, it returns a success response to the log sending module, along with an acknowledgment offset. After receiving the acknowledgment offset, the log sending module submits it to the offset management module. The offset management module then updates its local checkpoint and persists it to record the progress of the batch being successfully acknowledged by the server. Simultaneously, if the log management server's verification fails, it returns a failure response to the log sending module, carrying the expected offset. The log sending module notifies the offset management module to roll back the current physical read position to the position corresponding to the offset position. The corresponding position is determined, and batches that do not conform to the current offset state in the blocking queue are cleared. The file reading module then starts reading the log content again from the corrected position and resends it.
[0049] Optionally, in this embodiment, if the logical starting offset is inconsistent with the expected next offset, it is determined that the log batch has failed the continuity check. After the log management server generates a check failure response carrying the expected next offset and returns the check failure response to the log collection terminal, the method further includes: the log collection terminal receives the check failure response, queries the segment ledger according to the expected next offset, and reverses to locate the specified log file corresponding to the expected next offset and the specified position indicated by the expected next offset in the specified log file; the log collection terminal rolls back the current physical read position to the specified position and rolls back the global logical offset to the expected next offset; the log collection terminal cleans up the log batches to be sent in the local blocking queue whose logical starting offset is less than the expected next offset, or rearranges the remaining log batches to be sent in the blocking queue according to the logical offset order; the log collection terminal rereads the new newly added log data from the new current physical read position after the rollback and repackages it into a new log batch and sends it to the log management server.
[0050] In this embodiment, when the log management server detects an inconsistency between the logical starting offset and the expected next offset through continuity verification, the log management server determines that the log batch has failed the continuity verification. It then generates a verification failure response carrying the expected next offset and returns this response to the log collection end. It should be noted that this situation typically occurs when the logical starting offset in the log batch reported by the log collection end is not equal to the expected next offset maintained by the server for the corresponding log stream. This indicates a discrepancy between the collection progress recorded locally by the log collection end and the progress actually confirmed by the log management server. Possible causes of this discrepancy include the log collection end retransmitting old data already received by the log management server after an anomaly recovery; the log collection end's local checkpoint being lost or rolled back due to storage media failure or file corruption; expired batches accumulating in the blocking queue that the log collection end has not cleared in time; and the log management server having successfully received and confirmed some data, but an anomaly in the log collection end's confirmation response reception causing the local state to fail to update correctly. When any of the above situations occur, the log management server will send a clear correction instruction to the log collection terminal based on the expected next offset, so that the log collection terminal can perceive the state deviation and enter the correction process.
[0051] Upon receiving a verification failure response, the log sending module of the log collection terminal extracts the expected next offset from the response and initiates the offset correction process. The log collection terminal first queries the locally maintained segment ledger based on the expected next offset. Using the logical start and end addresses of each logical segment recorded in the segment ledger and their mapping relationship to physical files, it reverse-locates the target logical segment corresponding to the expected next offset. This determines the specified log file corresponding to the target logical segment and the specified position indicated by the expected next offset within that specified log file. Subsequently, the log collection terminal rolls back the current physical read position to this specified position and simultaneously rolls back the global logical offset to the expected next offset, ensuring that the local collection progress is aligned with the collection progress expected by the log management server at both the physical and logical levels. Meanwhile, the log collection terminal also needs to clean up or rearrange the log batches to be sent in the local blocking queue. Specifically, batches to be sent with a logical starting offset less than the expected next offset need to be discarded, because the log data corresponding to these batches has been confirmed as received by the log management server. For the remaining batches to be sent, they are rearranged according to the logical offset order to ensure that the log batches sent later remain ordered in the logical address space.
[0052] After completing the offset correction and blocking queue cleanup, the file reading module of the log collection terminal starts from the new current physical read position after the rollback, rereads the newly added log data, and repackages it into a new log batch according to the aforementioned encapsulation process, and sends it to the log management server again.
[0053] In this way, through the above feedback correction process, the log collection end can accurately align its local state with the server state under the explicit guidance of the log management server, thereby restoring to the correct collection progress. This ensures the order and continuity of the log data reported subsequently in the logical address space. Compared with the method of blindly rolling back based on the local state, the directional recovery based on the expected offset of the log management server proposed in this application embodiment can significantly improve the certainty and reliability of abnormal recovery.
[0054] For example, suppose the expected next offset of a log stream is 6300, while the logical starting offset of the log batch reported by the log collection end is 5800. The log management server returns a failure response carrying the expected next offset of 6300 after a discrepancy. Upon receiving this response, the log collection end queries the segment ledger to determine that the logical offset 6300 corresponds to byte offset 1500 in the app.log.1 file. It then rolls back the current physical read position to byte offset 1500 in app.log.1, rolls back the global logical offset to 6300, and discards old batches in the blocking queue with logical starting offsets less than 6300. It then starts reading again from this corrected position, repackages and sends new log batches, and resumes normal log collection after the log management server verifies the data.
[0055] In summary, the timing diagram for handling anomalies in the acquisition process described in the above embodiments is as follows: Figure 3 As shown, the interactive entities include the file reading module of the log collection terminal, the offset management module and the log sending module set in the log collection terminal, and the log management server. First, the file reading module sends a request to the offset management module to obtain the current reading position and the global logical offset. The offset management module returns the current file position and its corresponding offset to the file reading module. After obtaining the current read position, the file reading module reads the newly added log content from the local log file and encapsulates it into a LogBatch, then writes the LogBatch to the send queue. The log sending module retrieves the LogBatch from the send queue and sends it to the log management server. The log management server validates the batch; if validation fails, such as a continuity check, the log management server returns a failure response to the log sending module, containing an error code and the expected offset. Upon receiving a failure response, the log sending module notifies the offset management module to roll back the current physical read position and global logical offset to the position expected by the log management server. The offset management module updates the read position and logical offset accordingly. On the other hand, the log sending module cleans up or rearranges the batches to be sent in the blocking queue that do not conform to the current offset state to avoid repeating errors caused by continuing to send old batches. After the above correction is completed, the file reading module starts reading the log content again from the corrected offset position and repackages it into a new LogBatch for sending, thus completing the closed-loop correction process of the exception handling stage.
[0056] Optionally, updating and saving the current physical read position and global logical offset based on the confirmed offset includes: querying the segment ledger based on the confirmed offset, locating the target position corresponding to the confirmed offset in the target log file, updating the current physical read position to the target position, and updating the global logical offset to the confirmed offset; and saving the updated current physical read position and the updated global logical offset as local checkpoints.
[0057] In this embodiment, after receiving a successful verification response from the log management server, the log collection terminal extracts a confirmation offset from the response. This confirmation offset is the value obtained by the log management server after updating the expected next offset according to the logical end offset after passing the continuity verification. It represents the latest global logical offset position that the log management server has successfully processed, i.e., the current collection progress confirmed by the log management server. The log collection terminal queries the locally maintained segment ledger based on this confirmation offset. Using the logical start and end addresses of each logical segment recorded in the segment ledger and their mapping relationship with physical files, it reverse-locates the target position corresponding to the confirmation offset in the target log file. Specifically, the log collection end traverses the logical segment sequence belonging to the current log stream in the segment ledger, determines the target logical segment to which the confirmation offset belongs based on the logical start address and logical end address of each logical segment, and then calculates the specific byte offset position corresponding to the confirmation offset in the physical file, i.e. the target position, based on the physical file corresponding to the target logical segment and the difference between the confirmation offset and the logical start address of the logical segment. In this way, the confirmation offset in the logical address space is accurately mapped to the actual read point in the physical storage medium.
[0058] After location is established, the log collection client updates its current physical read position to the target position and its global logical offset to the confirmation offset. It then persistently saves the updated current physical read position and updated global logical offset as a local checkpoint to disk or non-volatile storage. By persistently saving the updated physical read position and logical offset, the log collection client can load the checkpoint from persistent storage after process restart or fault recovery, restoring the previously successfully confirmed physical read position and logical offset. Furthermore, the log collection client only advances the local checkpoint after receiving explicit success confirmation from the log management server, rather than committing it immediately when the log is read or placed in the queue. This ensures that the local checkpoint always represents the progress confirmed by the log management server.
[0059] In this way, through the reverse positioning and persistent storage process described above, the log collection end can convert the confirmation offset of the log management server into the actual executable read position within the physical file, so that the next collection task can continue from the position confirmed by the log management server. At the same time, the persistent local checkpoint provides crash recovery capability, ensuring that the process can still be restored from the last confirmed position after an unexpected exit, without having to re-collect the confirmed log data, thereby improving the reliability and resource utilization efficiency of the log collection system.
[0060] For example, assuming the log management server returns an acknowledgment offset of 6300, the log collection terminal queries the segment ledger and determines that this offset corresponds to the target log file app.log, and the logical starting address of this logical segment is 2000. Then, the calculated byte offset position of this acknowledgment offset within the app.log file is 4300. The log collection terminal updates the current physical read position to byte offset 4300 in the app.log file, updates the global logical offset to 6300, and persists the updated position information as a local checkpoint to the disk. Subsequent collection tasks will continue reading from byte offset 4300 in the app.log file.
[0061] Optionally, in this embodiment, the method further includes: after sending a log batch to the log management server, if a network timeout or connection failure is detected, the log collection terminal retains the current log batch to be sent and enters a retry process; in the retry process, the log batch is resent according to a preset retry strategy; if the retry process is successful, a verification success response returned by the log management server is received, the confirmation offset carried in the verification success response is extracted, and the current physical read position and global logical offset are updated and saved according to the confirmation offset to complete the management of the log stream; if the retry process fails after meeting the preset retry conditions, the acquisition of the log batch to be sent from the blocking queue is paused or the sending rate is reduced to create back pressure on the blocking queue and reduce the frequency of file reading; and / or, after receiving the log batch, the log management server performs basic verification on the log batch, the basic verification including at least source verification, format verification, log type verification, and offset verification. For continuous verification, after a log batch passes the basic verification, the log management server distributes the log batch to the corresponding write sub-channel according to the log type or priority. The log management server has multiple write sub-channels, each corresponding to a log type or priority. Each write sub-channel is independent of the others and has its own queue resources and consumption threads. When the log management server detects that the queue resources of any specified write sub-channel have reached the preset capacity limit, it triggers rate limiting for the log type or priority corresponding to the specified write sub-channel and returns a backpressure response to the specified log collection terminal that sent the log batch distributed to the specified write sub-channel. The backpressure response includes at least the current status of the specified write sub-channel and the suggested retry time, so that after receiving the backpressure response, the specified log collection terminal will pause or reduce the sending rate of logs of the corresponding log type or priority, and retain unacknowledged log batches for subsequent retries.
[0062] In this embodiment, after the log collection terminal sends the log batch to the log management server, if a network timeout or connection failure is detected—that is, if the log sending module in the log collection terminal encounters a connection timeout, connection reset, server unresponsiveness, DNS resolution failure, network jitter, or temporary server unavailability when sending the LogBatch to the server—the log collection terminal cannot determine whether the log management server has successfully received and processed the batch. Therefore, it cannot simply discard the batch, nor can it directly advance the local offset. The log collection terminal will retain the currently pending log batch and treat it as an "incompletely sent" batch, then enter the retry process. In the retry process, the log collection terminal will resend the log batch according to a preset retry strategy. This retry strategy can be one or more combinations of fixed-interval retries, exponential backoff retries, or setting a maximum number of retries, to avoid continuously high-frequency requests when the log management server fails or the network is unavailable, thus further increasing the system pressure. Due to a special case in network anomalies, such as the log management server actually successfully receiving and storing the batch, but the log collection end experiences a timeout or connection interruption before receiving the response, retrying to send may cause the same log batch to be received multiple times by the log management server. Therefore, if a retry is successful, the log management server returns a successful verification response and an acknowledgment offset. The log collection end extracts the acknowledgment offset and updates and persists it to the current physical read position and global logical offset to complete the management of the log batch. Subsequently, the log sending module continues to obtain subsequent batches from the blocking queue for sending.
[0063] If the retry process fails after meeting the preset retry conditions (i.e., multiple retries fail to obtain explicit success confirmation from the log management server), the log sending module in the log collection terminal will record network anomaly information and pause consuming new log batches from the blocking queue, or reduce the sending rate, waiting for network recovery. During this process, the file reading module in the log collection terminal may continue to write newly read log data to the blocking queue. When the blocking queue reaches its capacity limit, it will create backpressure on the file reading module, causing it to pause reading or reduce its reading frequency, thus preventing unlimited memory growth. In this case, since the log data is still retained in the local log file and the offset has not yet been committed, the log collection terminal can continue reading and sending from the last successfully confirmed offset position after the log collection system restarts or the network recovers. Therefore, network problems will not directly cause log loss, but only cause log synchronization delays. Thus, through the above retry and backpressure control mechanisms, the log collection system can ensure no data loss and no out-of-order offset commits under unstable communication conditions, providing the necessary prerequisite for backpressure control by the log management server. In summary, the timing diagram of the network problem and retry process described in the above embodiment is as follows: Figure 4As shown, the interactive entities include the file reading module of the log collection terminal, the offset management module and the log sending module set in the log collection terminal, and the log management server. First, the file reading module sends a request to the offset management module to obtain the current reading position and the global logical offset. The offset management module returns the current file position and its corresponding offset to the file reading module. After obtaining the current read position, the file reading module reads the local log content, encapsulates it into a LogBatch, and then writes it to the send queue. The log sending module retrieves the LogBatch from the send queue and sends it to the server. When the log sending module detects a network timeout or connection failure (i.e., no response from the server or a connection interruption), it retains the current LogBatch and enters a retry waiting state. It performs delayed retries according to a preset retry strategy, such as fixed intervals, exponential backoff, or a maximum retry limit, and then resends the same LogBatch to the server. If the retry is successful, the server returns a success response with an acknowledgment offset. The log sending module submits the confirmed offset to the offset management module, which then updates and persists the local checkpoint, completing the confirmation of this batch. If multiple retries fail, the log sending module pauses consuming new log batches from the blocking queue or reduces the sending rate. During this process, if the blocking queue reaches its capacity limit due to the log sending module pausing consumption, the full queue will create backpressure on the file reading module, forcing it to pause reading or reduce its reading frequency, thus preventing unlimited memory growth. Since the offset has not yet been committed at this stage, after a system restart or network recovery, reading and sending can continue from the last successfully confirmed offset, ensuring that log data is not lost in network problem scenarios.
[0064] Furthermore, the log management server needs to simultaneously receive concurrent data from multiple nodes, multiple collection agents, and various log types. Different logs have varying generation frequencies, data volumes, and importance. If all logs enter the same receiving queue and writing process, high-frequency logs will consume significant resources, thus blocking the processing of critical logs. To address this issue, in this embodiment, the log management server first performs basic verification after receiving a batch of logs. This basic verification includes at least source verification to confirm the legality of the log source, format verification to ensure the integrity and compliance of the log data structure, log type verification to identify the log category, and offset continuity verification to confirm that the logical offset is consistent with the server's expected state. After successful verification, the log management server distributes the logs to the corresponding writing sub-channels based on the log type or priority of the batch. The log management server has multiple writing sub-channels, each corresponding to a log type or priority. Each writing sub-channel is independent of the others and has its own independent queue resources and independent consumption threads. In this way, by classifying and isolating logs, we can prevent a certain type of high-frequency log from occupying management resources for a long time and affecting the reception and storage of other critical logs. This classification and isolation can be implemented in various forms, such as independent memory queues, independent persistent queues, independent message topics, independent partitions, or independent database write channels. For example, audit logs, execution logs, running logs, control logs, and system status logs can be written to different queues and processed by different consumer threads or write tasks. Audit logs can be set with higher priority and stricter acknowledgment policies, execution logs can use larger buffer queues and batch write strategies, and low-priority debug logs can have their reception frequency reduced when the system is under high pressure.
[0065] In addition to classification and isolation, the log management server in this embodiment also needs to perform rate limiting and backpressure control. When the log management server detects that the queue resources of any specified write sub-channel have reached the preset capacity limit, it triggers rate limiting for the log type or priority corresponding to the specified write sub-channel. For example, when a node continuously reports a large number of logs of a certain type, the log management server can limit only the log channel of that type without affecting the reporting of other types of logs. The log management server returns a backpressure response to the specified log collection terminal that sends the log batches distributed to the specified write sub-channel. The backpressure response includes at least the current status of the write sub-channel and the suggested retry time, and may further include information such as the server's expected offset. After receiving the backpressure response, the specified log collection terminal suspends or reduces the sending rate of logs of the corresponding log type or priority, and retains unacknowledged log batches for subsequent retries. In this way, through the combined design of classification and isolation, rate limiting, and backpressure control, the log management server can maintain stable operation under concurrent reporting from multiple nodes and multiple log types, avoiding the blocking of high-value critical logs by low-value, high-frequency logs. Furthermore, the collection end can reliably resume sending after rate limiting, congestion, or brief failures. For example, if the write sub-channel queue resources corresponding to audit logs reach their limit, the log management server triggers rate limiting for that type of log and returns a backpressure response to the corresponding collection agent. Upon receiving this response, the collection agent suspends sending batches of audit logs, retains unacknowledged logs, and waits for subsequent retries. Meanwhile, the collection and reporting of other log types, such as execution logs and runtime logs, remain unaffected and can continue normally.
[0066] In summary, the logical architecture of the technical solution in the embodiments of this application is as follows: Figure 5As shown, the log management system architecture in this embodiment is divided into four components in practical application: log collection agent, log management server, centralized storage, and management and display terminal. The core focus of this embodiment is the collaborative working mechanism between the log collection agent and the log management server. On the collection side, the log management system supports multiple log collection agents running simultaneously, such as Agent-i and Agent-j running simultaneously. Different agents can be deployed on different nodes or business servers and uniformly report log data to the log management server, thus forming a multi-agent distributed collection deployment pattern. Within each log collection agent, multiple collection threads can be started. For example, collection thread-1 is responsible for monitoring log path-1 and executing the corresponding file monitoring-1 and incremental reading-1, and collection thread-N is responsible for monitoring log path-N and executing the corresponding file monitoring-N and incremental reading-N, thereby realizing parallel collection of logs from multiple paths within a single node. The agent is also configured with an offset management module to save the reading progress of each log path, providing accurate positioning basis for file monitoring and incremental reading. The raw log content read by each collection thread is uniformly aggregated into the parsing and encapsulation module. This module organizes the raw log content into standardized log batches, attaching key metadata such as logical offset, log path, log type, node identifier, and proxy identifier to each batch. The log sending module then sends the encapsulated log batches to the log management server. On the management side, the log management server acts as a centralized receiving and processing center. It first verifies the format and offset information of the received log batches through the access verification module. Simultaneously, it maintains the expected offset status of each log stream based on the log stream identifier, i.e., the next logical offset the server expects to receive. After successful verification, the routing and distribution module distributes the log batches to different category write queues (e.g., category write queue 1, category write queue 2, up to category write queue N) based on dimensions such as proxy source, node information, log path, log type, or priority. Each category write queue is consumed by its corresponding write execution module and written to the backend centralized storage. Throughout the overall interaction, the server returns a success confirmation or expected offset value to the collection agent, which adjusts its local state accordingly. In addition, in practical applications, the log management server can also include a metadata storage module to record log-related metadata information.
[0067] In summary, the log management system architecture presents a distributed log management framework characterized by multi-agent data collection, centralized management on a single server, multi-threaded parallel monitoring within a single agent, and categorized writing to multiple queues on the server. The collection side is responsible for log collection and incremental reading on distributed nodes, while the management side is responsible for unified access verification, maintaining expected offset states according to log streams, routing and distributing logs by agent / node / path / type, isolating and executing writes to categorized queues, and ultimately achieving log data persistence through centralized storage. A unified visual operation and maintenance interface is provided through the management display. Through this architectural design, the log management system can adapt to complex scenarios of concurrent collection from multiple nodes, multiple log paths, and multiple log types in a distributed environment, and provides complete infrastructure support for upper-level mechanisms such as offset continuity verification, breakpoint resumption, categorized isolated writing, and server backpressure control.
[0068] The method provided in this application embodiment maintains a segment ledger for each log stream at the log collection end, mapping multiple log files generated by rotation to logical segments connected end-to-end on the logical address axis. This makes the collection progress no longer dependent on the specific log file identity. Therefore, when a log file is rotated, renamed, or truncated, the existing progress can still be mapped to the currently existing physical log file through the segment ledger, avoiding collection interruption caused by the invalidation of file identity. At the same time, the log management server independently maintains the expected next offset and performs continuity verification. The log collection end only advances and persists the local checkpoint after receiving the confirmed offset returned by the log management server, forming a closed-loop feedback mechanism with the log management server as the authoritative benchmark. Thus, when situations occur such as the log management server's memory progress has advanced but not yet been persisted, the log collection end can still recover from the last confirmed position without blindly rolling back, improving the certainty and reliability of abnormal recovery.
[0069] Furthermore, as Figure 1 In a specific implementation of the method, this application provides a distributed log management system based on global logical offsets, such as... Figure 6 As shown, the device includes: a log collection terminal 601 and a log management server 602.
[0070] The log collection terminal 601 is used to determine the currently managed log stream, locate the target log file of the current application in the log stream, and the log stream corresponds to a segment ledger for recording multiple logical segments connected end to end in chronological order. Each logical segment corresponds to a log file generated during the log rotation process. The log collection terminal 601 is also used to obtain the current physical read position of the target log file, and determine the global logical offset by combining the current physical read position and the segmented ledger. The current physical read position is used to locate the actual starting point of the log data to be read in the target log file, and the global logical offset is used to identify the continuous progress of the current physical read position in the log stream. The log collection terminal 601 is also used to read new log data from the target log file according to the actual starting point located by the current physical reading position, calculate the logical start offset and logical end offset according to the local offset of the new log data in the target log file and the global logical offset, encapsulate the new log data, the logical start offset and the logical end offset into a log batch, and send the log batch to the log management server. The log management server 602 is used to determine the expected next offset maintained for the log stream, and to perform continuity verification on the log batch by comparing the logical starting offset with the expected next offset. The log collection terminal 601 is also used to extract the confirmation offset carried in the verification success response when it receives the verification success response returned by the log management server, and update and save the current physical read position and the global logical offset according to the confirmation offset, so as to complete the management of the log stream.
[0071] In a specific application scenario, the log collection terminal 601 is used to query whether a target logical segment corresponding to the target log file exists in the segment ledger based on the file identity information of the target log file. The file identity information includes at least the path information and file system identification information of the target log file. If the target logical segment exists in the segment ledger, the logical starting address of the target logical segment is obtained, and the global logical offset is determined based on the sum of the logical starting address and the local offset of the current physical read position within the target log file. If the target logical segment does not exist in the segment ledger, the target log file is determined to be the first physical file appearing in the log stream. A new logical segment corresponding to the target log file is created, the logical starting address of the new logical segment is set to the current cumulative length value of the log stream, and the global logical offset is determined based on the sum of the logical starting address of the new logical segment and the local offset of the current physical read position within the target log file.
[0072] In a specific application scenario, the log collection terminal 601 is used to determine the actual starting point corresponding to the read start local offset by taking the local offset corresponding to the current physical read position within the target log file as the read start local offset; starting from the actual starting point, the newly added log data is read from the target log file according to preset encapsulation triggering conditions, wherein the encapsulation triggering conditions include one or more of the following: the amount of log data read reaches a preset size threshold, the number of log lines read reaches a preset number of lines threshold, and the reading duration reaches a preset time window; within the target log file, the minimum local offset corresponding to the newly added log data is determined as the start local offset, and the maximum local offset corresponding to the newly added log data within the target log file is determined as the end local offset; the logical start offset is calculated based on the sum of the global logical offset and the start local offset, and the logical end offset is calculated based on the sum of the global logical offset and the end local offset; The log collection terminal 601 includes a parsing and encapsulation module, which is used to encapsulate the number of new logs, the logical start offset, and the logical end offset to obtain the log batch.
[0073] In specific application scenarios, the log management server 602 includes: The verification module is used to obtain the expected next offset maintained for the log stream, the expected next offset being used to indicate the logical start offset of the expected next log batch; extract the logical start offset from the log batch, and compare whether the logical start offset is consistent with the expected next offset to determine whether the log batch has successfully passed the continuity verification; Accordingly, the verification module is further configured to: if the logical starting offset is consistent with the expected next offset, then determine that the log batch has successfully passed the continuity verification; the log management server updates the expected next offset according to the logical ending offset, uses the updated expected next offset as the confirmation offset, generates a verification success response carrying the confirmation offset, and returns the verification success response to the log collection terminal; if the logical starting offset is inconsistent with the expected next offset, then determine that the log batch has failed the continuity verification; the log management server generates a verification failure response carrying the expected next offset and returns the verification failure response to the log collection terminal.
[0074] In specific application scenarios, the log collection terminal 601 is also used to receive the verification failure response, query the segmented ledger according to the expected next offset, and reverse locate to the specified log file corresponding to the expected next offset and the specified position indicated by the expected next offset in the specified log file. The log collection terminal 601 is also used to roll back the current physical read position to the specified position, and to roll back the global logical offset to the expected next offset; The log collection terminal 601 is also used to clean up log batches to be sent in the local blocking queue whose logical starting offset is less than the expected next offset, or to rearrange the remaining log batches to be sent in the blocking queue according to the logical offset order. The log collection terminal 601 is also used to reread new newly added log data from the new current physical read position after rollback and repackage it into a new log batch to send to the log management server 602.
[0075] In specific application scenarios, the log collection terminal 601 also includes an offset management module. The offset management module is used to query the segmented ledger according to the confirmed offset, reverse locate the target position corresponding to the confirmed offset in the target log file, update the current physical read position to the target position, and update the global logical offset to the confirmed offset; and save the updated current physical read position and the updated global logical offset as local checkpoints.
[0076] In specific application scenarios, the log collection terminal 601 is further configured to, after sending the log batch to the log management server, if a network timeout or connection failure is detected, retain the currently pending log batch and enter a retry process; and in the retry process, resend the log batch according to a preset retry strategy; if the retry process is successful, receive the verification success response returned by the log management server, extract the confirmation offset carried in the verification success response, and update and save the current physical read position and the global logical offset based on the confirmation offset to complete the management of the log stream; if the retry process fails after meeting the preset retry conditions, pause the acquisition of the pending log batch from the blocking queue or reduce the sending rate to create back pressure on the blocking queue and reduce the frequency of file reading; and / or, The log management server 602 is further configured to perform basic verification on the log batch after receiving it. The basic verification includes at least source verification, format verification, log type verification, and offset continuity verification. After the log batch passes the basic verification, the log management server distributes the log batch to the corresponding write sub-channel according to the log type or priority corresponding to the log batch. The log management server has multiple write sub-channels, each corresponding to a log type or priority. Each write sub-channel is independent of the others and has independent... The log management server 602 has queue resources and independent consumption threads. When the queue resources of any specified write sub-channel reach the preset capacity limit, it triggers rate limiting for the log type or priority corresponding to the specified write sub-channel and returns a backpressure response to the specified log collection terminal that sends the log batches distributed to the specified write sub-channel. The backpressure response includes at least the current status of the specified write sub-channel and the suggested retry time, so that after receiving the backpressure response, the specified log collection terminal will pause or reduce the sending rate of logs of the corresponding log type or priority, and retain unacknowledged log batches for subsequent retries.
[0077] The apparatus provided in this application embodiment maintains a segment ledger for each log stream at the log collection end, mapping multiple log files generated by rotation to logical segments connected end-to-end on the logical address axis. This makes the collection progress no longer dependent on the specific log file identity. Therefore, when a log file is rotated, renamed, or truncated, the existing progress can still be mapped to the currently existing physical log file through the segment ledger, avoiding collection interruption caused by the invalidation of file identity identifiers. At the same time, the log management server independently maintains the expected next offset and performs continuity checks. The log collection end only advances and persists the local checkpoint after receiving the confirmed offset returned by the log management server, forming a closed-loop feedback mechanism with the log management server as the authoritative benchmark. Thus, when situations occur such as the log management server's memory progress has advanced but not yet been persisted, the log collection end can still recover from the last confirmed position without blindly rolling back, improving the certainty and reliability of abnormal recovery.
[0078] It should be noted that other corresponding descriptions of the functional units involved in the distributed log management system based on global logical offset provided in this application embodiment can be found in the following references. Figures 1 to 6 The corresponding descriptions in [the document] will not be repeated here.
[0079] The above embodiments and the technical features in the embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0080] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.
[0081] In an exemplary embodiment, see Figure 7 An electronic device is also provided, comprising a bus 701, a processor 702, a memory 703, and a communication interface 704. It may also include an input / output interface 705 and a display device 706. The various functional units can communicate with each other via the bus. The memory 703 stores a computer program, which, from bottom to top, includes a kernel 7034 that directly manages the hardware, middleware 7033 that provides general services for applications, an application programming interface 7032 that defines calling specifications, and an application 7031 that implements specific business logic. The processor 702 is used to read from the memory 703 via the bus 701 and execute the program stored in the memory, performing the distributed log management method based on global logical offsets described in the above embodiments.
[0082] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the distributed log management method based on global logical offsets.
[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented in hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0084] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.
[0085] Those skilled in the art will understand that the modules in the apparatus of the implementation scenario can be distributed within the apparatus of the implementation scenario as described, or they can be located in one or more apparatuses different from this implementation scenario, with corresponding changes. The modules of the above-described implementation scenario can be combined into one module, or they can be further divided into multiple sub-modules.
[0086] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenario.
[0087] The above disclosures are only a few specific implementation scenarios of this application. However, this application is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this application.
Claims
1. A distributed log management method based on global logical offsets, characterized in that, The method is applied to a log management system, which includes a log collection terminal and a log management server. The method includes: The log collection terminal determines the currently managed log stream, locates the target log file of the current application in the log stream, and the log stream corresponds to a segment ledger for recording multiple logical segments connected end to end in chronological order. Each logical segment corresponds to a log file generated during the log rotation process. The log collection terminal obtains the current physical read position of the target log file, and determines the global logical offset by combining the current physical read position with the segmented ledger. The current physical read position is used to locate the actual starting point of the log data to be read in the target log file, and the global logical offset is used to identify the continuous progress of the current physical read position in the log stream. The log collection terminal reads new log data from the target log file according to the actual starting point located by the current physical reading position. Based on the local offset of the new log data in the target log file and the global logical offset, it calculates the logical start offset and logical end offset, encapsulates the new log data, the logical start offset, and the logical end offset into a log batch, and sends the log batch to the log management server. The log management server determines the expected next offset for the log stream maintenance, and performs continuity verification on the log batch by comparing the logical starting offset with the expected next offset. Upon receiving a successful verification response from the log management server, the log collection terminal extracts the confirmation offset carried in the successful verification response, and updates and saves the current physical read position and the global logical offset based on the confirmation offset to complete the management of the log stream.
2. The method according to claim 1, characterized in that, The step of obtaining the current physical read position of the target log file, and determining the global logical offset by combining the current physical read position with the segmented ledger, includes: Based on the file identity information of the target log file, query whether there is a target logical segment in the segment ledger that corresponds to the target log file. The file identity information includes at least the path information and file system identification information of the target log file. If the target logical segment exists in the segmented ledger, the logical starting address of the target logical segment is obtained, and the global logical offset is determined based on the sum of the logical starting address and the local offset of the current physical read position in the target log file. If the target logical segment does not exist in the segment ledger, the target log file is determined to be the first physical file to appear in the log stream. A new logical segment is created for the target log file, and the logical starting address of the new logical segment is set to the current cumulative length value of the log stream. The global logical offset is determined based on the sum of the logical starting address of the new logical segment and the local offset of the current physical read position within the target log file.
3. The method according to claim 1, characterized in that, The process involves reading new log data from the target log file based on the actual starting point located according to the current physical read position, calculating logical start offset and logical end offset based on the local offset of the new log data within the target log file and the global logical offset, and encapsulating the new log data, the logical start offset, and the logical end offset into a log batch, including: The local offset corresponding to the current physical read position within the target log file is used as the read start local offset to determine the actual starting point corresponding to the read start local offset. Starting from the actual starting point, the newly added log data is read from the target log file according to the preset encapsulation triggering conditions, wherein the encapsulation triggering conditions include one or more of the following: the amount of log data read reaches a preset size threshold, the number of log lines read reaches a preset number of lines threshold, and the reading duration reaches a preset time window. Within the target log file, the minimum local offset corresponding to the newly added log data is determined as the starting local offset, and the maximum local offset corresponding to the newly added log data within the target log file is determined as the ending local offset. The logical start offset is calculated based on the sum of the global logical offset and the start local offset, and the logical end offset is calculated based on the sum of the global logical offset and the end local offset. The newly added log number, the logical start offset, and the logical end offset are encapsulated to obtain the log batch.
4. The method according to claim 1, characterized in that, The determination of the expected next offset for the log stream maintenance involves comparing the logical starting offset with the expected next offset to perform continuity verification on the log batch, including: Obtain the expected next offset maintained for the log stream, the expected next offset being used to indicate the logical start offset of the next log batch expected to be received; Extract the logical starting offset from the log batch and compare whether the logical starting offset is consistent with the expected next offset to determine whether the log batch has successfully passed the continuity check. Accordingly, the method further includes: If the logical starting offset is consistent with the expected next offset, then the log batch is determined to have successfully passed the continuity check. The log management server updates the expected next offset according to the logical ending offset, and uses the updated expected next offset as the confirmation offset. It generates a check success response carrying the confirmation offset and returns the check success response to the log collection terminal. If the logical starting offset is inconsistent with the expected next offset, it is determined that the log batch has failed the continuity check. The log management server generates a check failure response carrying the expected next offset and returns the check failure response to the log collection terminal.
5. The method according to claim 4, characterized in that, If the logical starting offset is inconsistent with the expected next offset, then the log batch has failed the continuity check. After the log management server generates a check failure response carrying the expected next offset and returns the check failure response to the log collection terminal, the method further includes: The log collection terminal receives the verification failure response, queries the segmented ledger according to the expected next offset, and reverses the location to the specified log file corresponding to the expected next offset and the specified position indicated by the expected next offset in the specified log file; The log collection terminal rolls back the current physical read position to the specified position and rolls back the global logical offset to the expected next offset; The log collection terminal cleans up log batches to be sent in the local blocking queue whose logical starting offset is less than the expected next offset, or rearranges the remaining log batches to be sent in the blocking queue according to the logical offset order. The log collection terminal starts reading new log data from the new current physical read position after the rollback and repackages it into a new log batch before sending it to the log management server.
6. The method according to claim 1, characterized in that, The step of updating and saving the current physical read position and the global logical offset based on the confirmed offset includes: Based on the confirmed offset, query the segmented ledger, reverse locate the target position corresponding to the confirmed offset in the target log file, update the current physical read position to the target position, and update the global logical offset to the confirmed offset; The updated current physical read position and the updated global logical offset are saved as local checkpoints.
7. The method according to claim 1, characterized in that, The method further includes: After sending the log batch to the log management server, if a network timeout or connection failure is detected, the log collection terminal retains the current log batch to be sent and enters a retry process. In the retry process, the log batch is resent according to a preset retry strategy. If the retry process is successful, the terminal receives a successful verification response from the log management server, extracts the acknowledgment offset carried in the successful verification response, and updates and saves the current physical read position and the global logical offset based on the acknowledgment offset to complete the management of the log stream. If the retry process fails after meeting preset retry conditions, the terminal pauses the retrieval of log batches to be sent from the blocking queue or reduces the sending rate to create backpressure on the blocking queue, reducing the frequency of file reading; and / or, After receiving the log batch, the log management server performs basic checks on the log batch, including at least source verification, format verification, log type verification, and offset continuity verification. After the log batch passes the basic checks, the log management server distributes the log batch to the corresponding write sub-channel according to the log type or priority. The log management server has multiple write sub-channels, each corresponding to a log type or priority. Each write sub-channel is independent of the others and has independent queue resources and independent consumption threads. When the log management server detects that the queue resources of any specified write sub-channel have reached a preset capacity limit, it triggers rate limiting for the log type or priority corresponding to the specified write sub-channel and returns a backpressure response to the specified log collection terminal that sent the log batch distributed to the specified write sub-channel. The backpressure response includes at least the current status of the specified write sub-channel and a suggested retry time, so that after receiving the backpressure response, the specified log collection terminal pauses or reduces the sending rate of logs of the corresponding log type or priority and retains unacknowledged log batches for subsequent retries.
8. A distributed log management system based on global logical offsets, characterized in that, The log management system includes a log collection terminal and a log management server: The log collection terminal is used to determine the currently managed log stream, locate the target log file of the current application in the log stream, and the log stream corresponds to a segment ledger for recording multiple logical segments connected end to end in chronological order. Each logical segment corresponds to a log file generated during the log rotation process. The log collection terminal is also used to obtain the current physical read position of the target log file, and determine the global logical offset by combining the current physical read position and the segmented ledger. The current physical read position is used to locate the actual starting point of the log data to be read in the target log file, and the global logical offset is used to identify the continuous progress of the current physical read position in the log stream. The log collection terminal is also used to read new log data from the target log file according to the actual starting point located by the current physical reading position, calculate the logical start offset and logical end offset according to the local offset of the new log data in the target log file and the global logical offset, encapsulate the new log data, the logical start offset and the logical end offset into a log batch, and send the log batch to the log management server. The log management server is used to determine the expected next offset to be maintained for the log stream, and to perform continuity verification on the log batch by comparing the logical starting offset with the expected next offset. The log collection terminal is also used to extract the confirmation offset carried in the verification success response when it receives the verification success response returned by the log management server, and update and save the current physical read position and the global logical offset according to the confirmation offset, so as to complete the management of the log stream.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.
10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.
