A log management method

Abstract

This invention discloses a log management method. During the execution of a front-end business thread, generated log information is written to a pre-created circular linked list containing at least two shared memory segments linked bidirectionally by pointers. This achieves high-speed buffering of log data and decoupling of low-latency business thread operations. Simultaneously, a multi-dimensional asynchronous persistence mechanism is constructed by writing the shared memory segment containing log information to disk when any of several preset conditions are met, including a full segment in the circular linked list, timeout failure to write, receiving a thread termination signal, or process crash. This method improves throughput through batch writing in normal high-concurrency scenarios, ensures real-time performance through a timeout mechanism in low-frequency scenarios, and provides emergency data recovery in case of abnormal exits or crashes. Therefore, it significantly improves log reliability and anomaly recovery capabilities while reducing business latency.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a log management method. Background Technology

[0002] Existing logging systems typically employ synchronous disk writing or simple asynchronous queues. When services crash, experience sudden spikes in traffic, or processes exit abnormally, buffered logs are easily lost, making it difficult to reproduce the fault. Under multi-threaded contention, disk write jitter is significant, impacting business latency and lacking continuity across restarts. Log rotation is prone to overwriting or becoming out of order, resulting in insufficient operational observability. Therefore, achieving low-latency, highly reliable log persistence in high-concurrency scenarios has become a pressing technical challenge. Summary of the Invention

[0003] This application provides a log management method that solves the technical problems in the prior art where log data is easily lost and real-time performance is difficult to guarantee when services exit abnormally or experience sudden peaks. It achieves the technical effect of significantly improving log reliability and anomaly recovery capability while reducing business latency through a shared memory ring buffer and an asynchronous persistence mechanism triggered by multiple conditions.

[0004] This application provides a log management method, the method comprising: During the execution of the front-end business thread, the log information generated by the front-end business thread is written into the free shared memory segment of a pre-created circular linked list. The circular linked list is composed of at least two shared memory segments linked bidirectionally by pointers. If at least one of the preset conditions is met, the log information in at least one shared memory segment containing log information in the circular linked list is written to the disk. The preset conditions include the existence of a full shared memory segment in the circular linked list, a first time difference between the current time and the time when log information was last written to the disk exceeding a first preset time window, receiving a thread termination signal, and the process in which the front-end business thread is located crashing.

[0005] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: This application's embodiments achieve high-speed buffering of log data and low-latency decoupling of business threads by writing generated log information into the free shared memory segment of a pre-created circular linked list consisting of at least two shared memory segments linked bidirectionally by pointers during the execution of the front-end business thread. Simultaneously, by writing the shared memory segment containing log information to disk when any one of several preset conditions is met, including a full segment in the circular linked list, timeout without writing, receiving a thread termination signal, or process crash, a multi-dimensional triggered asynchronous persistence mechanism is constructed. This method improves throughput through batch writing in normal high-concurrency scenarios, ensures real-time performance through a timeout mechanism in low-frequency scenarios, and urgently recovers data in case of abnormal exit or crash, thereby significantly improving log reliability and anomaly recovery capabilities while reducing business latency. Attached Figure Description

[0006] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0007] Figure 1 A flowchart illustrating a log management method provided in an embodiment of this application; Figure 2 A schematic diagram of a log system architecture provided in an embodiment of this application; Figure 3 This application provides a schematic diagram of a front-end log writing process. Figure 4 This application provides a schematic diagram of a backend log persistence process. Figure 5 This is a schematic diagram of a log file rotation process provided in an embodiment of this application. Detailed Implementation

[0008] This application provides a log management method that solves the technical problems in the prior art where log data is easily lost and real-time performance is difficult to guarantee when services exit abnormally or experience sudden peaks. It achieves the technical effect of significantly improving log reliability and anomaly recovery capability while reducing business latency through a shared memory ring buffer and an asynchronous persistence mechanism triggered by multiple conditions.

[0009] The technical solution of this application embodiment is to solve the above-mentioned technical problems, and the general idea is as follows: This application's embodiments achieve high-speed buffering of log data and low-latency decoupling of business threads by writing generated log information into the free shared memory segment of a pre-created circular linked list consisting of at least two shared memory segments linked bidirectionally by pointers during the execution of the front-end business thread. Simultaneously, by writing the shared memory segment containing log information to disk when any one of several preset conditions is met, including a full segment in the circular linked list, timeout without writing, receiving a thread termination signal, or process crash, a multi-dimensional triggered asynchronous persistence mechanism is constructed. This method improves throughput through batch writing in normal high-concurrency scenarios, ensures real-time performance through a timeout mechanism in low-frequency scenarios, and urgently recovers data in case of abnormal exit or crash, thereby significantly improving log reliability and anomaly recovery capabilities while reducing business latency.

[0010] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0011] First, it should be clarified that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0012] With the increasing prevalence of high concurrency and complex business scenarios in server-side and embedded systems, logs have become a core basis for fault location, performance analysis, and compliance auditing. However, traditional synchronous disk writing or simple queue asynchronous disk writing solutions have revealed a series of shortcomings when dealing with abnormal service crashes, abnormal process exits, and sudden traffic spikes: First, crashes lead to the loss of a large amount of data that has not been written to disk in the buffer, making it difficult to reproduce the context at the last moment; second, disk writing operations fluctuate wildly under multi-threaded contention, severely dragging down the response latency of business threads; in addition, log rotation mechanisms lack continuity across restarts, which can easily cause file overwriting or disordered order, affecting subsequent retrieval efficiency; finally, insufficient observability at the operational level makes it impossible to monitor buffer occupancy, log discarding behavior, and disk writing health in real time, making it difficult to discover and locate system bottlenecks in a timely manner.

[0013] To address the aforementioned problems, this application provides a log management method, which includes steps S11-S12, as detailed below. Figure 1 As shown.

[0014] Step S11: During the operation of the front-end business thread, the log information generated by the front-end business thread is written into the free shared memory segment of the pre-created circular linked list. The circular linked list is composed of at least two shared memory segments linked bidirectionally by pointers. Step S12: If at least one of the preset conditions is met, write the log information in at least one shared memory segment containing log information in the circular linked list to the disk. The preset conditions include: the existence of a full shared memory segment in the circular linked list; the first time difference between the current time and the time when log information was last written to the disk exceeds a first preset time window; receiving a thread termination signal; and the process where the front-end business thread is located crashes.

[0015] This application provides a log management method that can be executed by any electronic device with data processing capabilities. The electronic device can be a server, desktop computer, laptop computer, tablet computer, embedded device, or vehicle terminal, etc. This device typically includes a processor, memory, and a communication interface. The processor executes program instructions stored in the memory to implement the log management method described in this application. In specific implementation, the device runs an operating system and deploys business applications that require logging. During operation, the business thread calls the log interface, and the log management module writes log information into a shared memory circular linked list according to the method of this application. When certain conditions are met, a persistent thread writes the log information to disk. Those skilled in the art will understand that the above device is merely an example, and other devices with computing and storage capabilities are also applicable to this application.

[0016] Before describing steps S11 and S12, this application embodiment will first describe the system architecture and tools corresponding to the method provided in this application embodiment.

[0017] This application provides a log management method that can be implemented based on a specially designed log system architecture. Please refer to... Figure 2 The core modules and their functions of the system are described below.

[0018] The system employs an in-process singleton logging engine for unified management. This logging engine is created when the process starts, and only one instance exists throughout the entire process. All logging operations are performed through this singleton to avoid resource conflicts and data inconsistencies. The engine primarily maintains a doubly linked circular list stored in shared memory, consisting of multiple shared memory nodes called "segments" (cell_buffer). Each segment has a fixed capacity and its availability is managed through two status flags: "FREE" and "FULL".

[0019] At the log writing level, the system adopts a front-end and back-end separation architecture. The front-end consists of application business threads, which record logs using system-provided log macros (such as LOG_TRACE, LOG_DEBUG, LOG_INFO, LOG_ERROR, etc.) or the try_append() function interface. The business threads are only responsible for formatting log information into standard strings and attempting to append those strings to the currently available shared memory segment. This entire process does not involve any disk I / O operations, thus minimizing the latency impact on the business threads. The back-end is a separate persistence thread (persist()), which is responsible for batch writing segments that are full in shared memory to disk. Furthermore, to avoid data lingering in memory in low-frequency scenarios with small log volumes, this thread is configured with a timeout policy. If the wait exceeds a preset time window, it will be forcibly woken up and flushed to disk to ensure log real-time performance.

[0020] The shared memory segments serve as temporary carriers of log data. Each segment is created upon process startup, and its header stores not only a forward pointer (prev) and a backward pointer (next), but also its own shared memory identifier (shmid). These pointers connect multiple segments into a doubly linked circular list, facilitating system traversal and segment switching. The independent shmid allows external maintenance tools to attach to each segment independently. During operation, when a front-end thread fills a segment, it sets the segment's state to FULL and switches to the next segment via the next pointer to continue writing. When a back-end thread persists the segment's data to disk, it executes a clear() operation to clear the data within the segment and restores its state to FREE, allowing the front-end thread to reuse it.

[0021] The system also includes a log rotation module specifically for managing log files on disk. This module uses a naming convention of "date + incrementing sequence number," specifically in the format {dir} / {prog}.{YYYY}{MM}{DD}.log.{idx}, where idx is obtained by taking the global sequence number (seq) modulo 100, ensuring that filenames generated within the same day are unique and consecutive. To support continuity across restarts, the system uses a metadata file to persistently store the current global sequence number (seq), allowing processes to continue incrementing the sequence number after restarting, effectively preventing file overwriting. Furthermore, the number of log files is configurable; when the total number of files exceeds a preset limit, the system automatically cleans up the oldest files based on their sequence number to control disk space usage.

[0022] To improve system observability and ease of operation and maintenance, this application provides supporting operation and maintenance tools. Among them, the `flusher` tool can run outside the logging process. It iterates through the segments in shared memory and exports any residual data that has not yet been written to disk to a specified file. With the optional `-d` parameter, the tool can also clean up and release shared memory space after exporting the data. The `cell_monitor` tool can periodically display detailed status information for each shared memory segment, including its status (FREE / FULL), total size, used size, available size, and predecessor and successor pointers, thereby assisting operation and maintenance personnel in capacity assessment and health diagnosis.

[0023] Finally, the system incorporates signal handling and exception recovery mechanisms. The system registers simplified handling functions for critical signals such as SIGTERM, SIGQUIT, and SIGSEGV. These functions adhere to asynchronous signal safety principles, performing only the simplest operations, such as writing a single byte to a dedicated pipe to wake up the backend persistent thread, without performing complex I / O operations. When a process is about to terminate due to a crash, the backend thread, once awakened, can immediately initiate an emergency disk flushing process, writing all log data in shared memory that has not yet been written to disk (especially the logs from the last moment before the crash) to the disk, thereby maximizing data recovery. Furthermore, the system provides an optional compiler macro, ENABLE_RINGLOG_HISTORY_EXPORT. When this macro is enabled, new processes automatically detect and export historical data remaining in shared memory from the previous crash to a specified path upon startup, and then clean it up to prevent residual data from overwriting new logs.

[0024] Based on the above system architecture, this application provides a log management method, which will be described below in conjunction with the appendix. Figure 2 Steps S11 and S12 of the embodiments of this application will be described in detail.

[0025] Regarding step S11, during the operation of the front-end business thread, the log information generated by the front-end business thread is written into the free shared memory segment of a pre-created circular linked list. The circular linked list is composed of at least two shared memory segments linked bidirectionally by pointers.

[0026] In this embodiment, the logging environment needs to be initialized during system startup. Specifically, when the process starts, the fixed capacity (e.g., 64KB) of each shared memory segment is set by calling LOG_MEM_SET(bytes), laying the foundation for subsequent memory allocation. Then, LOG_INIT(log_dir, prog_name, level) is called to initialize the logging system: this function constructs the file path based on the specified log directory and program name, creates a backend persistent thread and sets it to a detached state, and generates stable shared memory key values ​​using rlog_ftok(SHM_NAME, SHM_KEY_ID_SEQ), thereby creating or obtaining the main shared memory region and a doubly linked circular list composed of multiple shared memory segments. Each shared memory segment contains forward and backward pointers, which link the segments into a ring (the forward pointer of the first shared memory segment is linked to the backward pointer of the last shared memory segment), and record their respective shared memory identifiers (shmid) for subsequent access and management within and outside the process. After initialization, the system is in a write-ready state.

[0027] Based on the aforementioned initialization, in step S11, when the front-end business thread runs and generates log information, the thread writes the formatted log data into a free shared memory segment of a pre-created circular linked list. Formatting refers to concatenating structured log information (level, time, file, line number, message, etc.) into a complete, human-readable text line. The circular linked list consists of at least two shared memory segments (which can be simply referred to as segments) linked bidirectionally by pointers, ensuring that multiple segments logically constitute a reusable buffer pool. Through this mechanism, the business thread only performs lightweight memory copy operations, without involving disk I / O or locking waits, thereby achieving low-latency, low-intrusion log recording for the business thread, significantly improving the overall throughput of the system, and laying the data foundation for subsequent asynchronous persistence.

[0028] Specifically, before writing the log information generated by the front-end business thread into the free shared memory segment of the pre-created circular linked list, the method further includes: Determine whether the remaining space of the current shared memory segment pointed to by the current write pointer of the front-end business thread is greater than or equal to the first space required by the first log information to be written. If the remaining space of the current shared memory segment is greater than or equal to the first space, then the current shared memory segment is determined to be a free shared memory segment; If the remaining space of the current shared memory segment is less than the first space, the current shared memory segment is determined as a full shared memory segment, and the remaining space of each shared memory segment is judged in turn according to the order of each shared memory segment in the circular linked list to see if it is greater than or equal to the first space, until a shared memory segment with a remaining space greater than or equal to the first space is found, and it is regarded as an idle shared memory segment.

[0029] In this embodiment, before writing logs, the front-end business thread first needs to determine the currently available shared memory segment through a judgment mechanism. Specifically, each front-end business thread maintains a "current write pointer," which points to the shared memory segment used by its last write operation (i.e., the "current shared memory segment"). When new log information (referred to as "first log information") needs to be written, the thread first obtains the size of the bytes required after formatting the first log information (referred to as "first space"), and then determines whether the remaining space in the current shared memory segment is sufficient to accommodate the first space.

[0030] If the remaining space in the current shared memory segment is greater than or equal to the first space, it means that the segment is still available. The thread identifies it as a "free shared memory segment" and directly writes the first log information into it, while updating the write pointer position within the segment. If the remaining space in the current shared memory segment is less than the first space, it means that the segment cannot accommodate new logs. In this case, the thread marks the state of the segment as a "full shared memory segment" and obtains the next shared memory segment in the circular linked list through the "next" pointer stored at the head of the segment. It checks the remaining space of each subsequent segment in turn until it finds a segment with remaining space greater than or equal to the first space. It identifies this segment as the new "free shared memory segment," updates the current write pointer to point to this segment, and performs the write operation. This process ensures that log data always fills each segment sequentially according to the order of the circular linked list, avoiding management chaos caused by random writes. At the same time, the "check first, write later" mechanism ensures the safety and efficiency of memory operations.

[0031] For example, suppose a circular linked list contains three shared memory segments A, B, and C, each 64KB in size. The current write pointer points to segment A, which has already written 50KB of data, leaving 14KB of space. At this time, a front-end business thread generates a log entry that, after formatting, requires 10KB of space (the first 10KB). The thread first checks that the remaining space in segment A is 14KB ≥ 10KB, so it directly writes the log to segment A, increasing the used space to 60KB, completing the write. If another log entry subsequently requires 20KB of space, and segment A has only 4KB of remaining space (less than 20KB), the thread marks segment A as "full" and uses the next pointer of segment A to find segment B. Assuming segment B is currently free, with 64KB ≥ 20KB of remaining space, the thread identifies segment B as the new free segment, updates the current write pointer to point to segment B, and writes the log to segment B. Through this mechanism, the logging system achieves ordered and efficient utilization of the circular buffer while minimizing the latency impact on business threads.

[0032] Furthermore, after determining whether the remaining space of each shared memory segment is greater than or equal to the first space according to the order of each shared memory segment in the circular linked list, the method further includes: If the remaining space of each shared memory segment in the circular linked list is less than the first space, then determine whether the memory size currently occupied by the circular linked list is greater than or equal to the preset memory limit corresponding to the circular linked list. If the memory currently occupied by the circular linked list is less than the preset memory limit, a new shared memory segment is created in the circular linked list, and the newly created shared memory segment is used as an idle shared memory segment. If the memory currently occupied by the circular linked list is greater than or equal to the preset memory limit, the first log information to be written will be discarded and the time of this discard will be recorded.

[0033] In this embodiment, after traversing all shared memory segments in the order of the circular linked list, if the remaining space of each segment is found to be less than the space required for the current log to be written (i.e., the first space), it indicates that the circular linked list is currently full or about to be full. At this time, the system does not immediately discard the logs, but first performs a capacity check: it obtains the total memory occupied by all shared memory segments in the current circular linked list (i.e., the number of segments created multiplied by the size of a single segment) and compares it with the system's preset "memory limit." This memory limit is configured during the initialization phase to control the maximum shared memory resources that the log system can occupy, preventing the stability of the business process from being affected by the unlimited growth of the log buffer.

[0034] If the current total memory usage is less than the preset memory limit, it means the system still has room for expansion. In this case, a new shared memory segment is dynamically created in the circular linked list. The newly created segment is inserted into the appropriate position in the circular linked list (e.g., after the current write pointer), and is integrated into the circular structure by adjusting the forward and backward pointers of adjacent segments. Then, this new segment is used as a "free shared memory segment" for current log writing. This dynamic expansion mechanism allows the system to elastically increase its buffer capacity during sudden traffic spikes, avoiding the loss of a large number of logs due to insufficient fixed segments.

[0035] Conversely, if the current total memory usage has reached or exceeded the preset memory limit, it means that the system can no longer accommodate more logs through expansion. In this case, the first log message to be written is discarded, and the time of this discard is recorded internally by the system (e.g., saved as "last_drop_time"). This discarding behavior is part of a lossy degradation strategy, the purpose of which is to protect business threads from the impact of the log system being overloaded, and at the same time, to provide a basis for subsequent rate limiting control by recording the discard time.

[0036] For example, suppose the preset memory limit is 256KB and the segment size is 64KB, so a maximum of 4 segments can be created (256KB ÷ 64KB = 4). The current circular linked list already contains 4 segments A, B, C, and D, and the remaining space in each segment is insufficient to accommodate a new 10KB log entry (e.g., each segment has already written more than 55KB). At this point, the system checks the current memory usage: 4 segments × 64KB = 256KB, which has reached the memory limit, so no new segment can be created. Therefore, the system discards this 10KB log entry and records the current time (e.g., 10:00:00) in memory as the time of this discard. This mechanism ensures that under extreme full load conditions, system behavior is predictable and resource usage is controlled, while providing a data foundation for subsequent log loss analysis and rate limiting strategies.

[0037] Furthermore, after discarding the first log information to be written and recording the time of this discard, the method further includes: If the next log message to be written is received, determine whether the second time difference between the current time and the recorded time of this discard is greater than or equal to the second preset time window; If the second time difference is less than the second preset time window, the second log information is discarded, and the time of this discard is updated. If the second time difference is greater than or equal to the second preset time window, then the remaining space of each shared memory segment in the circular linked list is determined sequentially according to the order of each shared memory segment to see if it is greater than or equal to the second space required by the second log information. If the remaining space of each shared memory segment in the circular linked list is less than the second space, the second log information is discarded and the discard time is updated; if there is a shared memory segment in the circular linked list with a remaining space greater than or equal to the second space, the shared memory segment is determined as an idle shared memory segment and the second log information is written to the idle shared memory segment.

[0038] In this embodiment, when the system discards a log because there is not enough space in all shared memory segments and records the time of the discard, the subsequent log processing will introduce a "rate limiting and suppression" mechanism to prevent a time-consuming full linked list traversal check from being performed for each log when the system is continuously under full load, thereby further aggravating the system load.

[0039] Specifically, after the system records the moment of the first log drop (referred to as the "current drop moment"), if new logs to be written are subsequently received (referred to as the "second log information"), the system first calculates the time difference between the current moment and the recorded current drop moment (referred to as the "second time difference"), and compares this time difference with a preset "second preset time window" (e.g., 1 second). If the second time difference is less than the second preset time window, it indicates that the system recently experienced log drop and is likely still under full load. In this case, to avoid repeatedly performing expensive full linked list scans, the system adopts a fast degradation strategy: directly discarding the second log information and updating the current drop moment to the current moment. This operation does not involve any linked list traversal or space checks, greatly reducing interference with business threads.

[0040] If the second time difference is greater than or equal to the second preset time window, it indicates that a sufficiently long time has passed since the last discard, and the system may have eased its load (for example, backend threads may have released some free segments). At this point, the system re-executes the complete space check process: sequentially checking whether the remaining space of each shared memory segment is sufficient to accommodate the second space required for the second log information, according to the order of the circular linked list. If, after traversal, it is found that the remaining space of all segments is still less than the second space, it indicates that the system is still in a fully loaded state. At this point, the second log information is discarded, and the discard time is updated. If a segment with sufficient remaining space is found during the traversal, the segment is identified as a free segment, the second log information is written into it, and the pointers within the segment are updated normally.

[0041] For example, suppose the second preset time window is set to 1 second. At 10:00:00, the system discards a log entry because all segments are full, recording the discard time as 10:00:00. Then, at 10:00:00.5 (only 0.5 seconds from the last discard, less than 1 second), a new log entry arrives, which the system discards immediately, updating the discard time to 10:00:00.5. Next, at 10:00:01.6 (1.1 seconds from the last discard, greater than 1 second), another log entry arrives. At this point, the system performs a complete space check: after traversing all segments, it finds that segment B has free space (the backend thread has flushed it to disk and set it to FREE), so it writes this log entry to segment B, resuming normal writing. Through this mechanism, the system avoids high-frequency invalid traversals during continuous full load, significantly reducing processor overhead. Simultaneously, it can automatically resume writing after the load eases, achieving smooth overload protection and self-recovery capabilities.

[0042] Regarding step S12, if at least one of the preset conditions is met, the log information in at least one shared memory segment containing log information in the circular linked list is written to the disk. The preset conditions include the existence of a full shared memory segment in the circular linked list, the first time difference between the current time and the time when log information was last written to the disk exceeding a first preset time window, receiving a thread termination signal, and the process where the front-end business thread is located crashing.

[0043] This application embodiment achieves reliable writing of log data from a shared memory circular buffer to disk through a multi-condition triggered asynchronous persistence mechanism. Specifically, when any one of the preset conditions is met, the system triggers the backend persistence thread to write one or more shared memory segments containing log information from the circular linked list to disk.

[0044] The set of preset conditions includes four typical scenarios: When a shared memory segment in the circular linked list is full, a batch write is triggered to fully utilize the disk I / O throughput. When the time difference between the current moment and the moment when log information was last written to the disk exceeds the first preset time window, a timeout write is triggered to prevent logs from remaining in memory for too long in low-frequency scenarios. When a thread termination signal is received or the process where the front-end business thread is located is detected to have crashed, an emergency write is triggered to salvage the log data that has not yet been written to disk to the greatest extent possible.

[0045] Through this mechanism, the embodiments of this application achieve the following technical effects: First, the multi-condition triggering design balances high throughput and low latency, improving efficiency through batch writing in high-concurrency scenarios and ensuring real-time performance through a timeout mechanism in low-frequency scenarios; Second, emergency writing in abnormal situations significantly reduces the probability of log loss due to process crashes, preserving critical context for fault recovery; Finally, by separating log writing operations from business threads and having them handled uniformly by an independent backend thread, business threads are prevented from being blocked by disk I / O, achieving low-intrusion, high-performance log recording.

[0046] In the first typical scenario, if there is a full shared memory segment in the circular linked list, the log information in the full shared memory segment is written to the disk.

[0047] In this embodiment, when a shared memory segment in the circular linked list is full, the system will trigger the backend persistence thread to write the log information in these full segments to the disk in batches. This mechanism corresponds to the main write mode in high-throughput scenarios, and its core lies in achieving efficient collaboration between the frontend business thread and the backend persistence thread through condition variables.

[0048] Specifically, during log writing, if the front-end business thread finds that the currently used shared memory segment has reached its capacity limit after writing a certain log entry, or needs to switch segments due to insufficient remaining space, it will mark the segment's state from "free" to "full". Subsequently, the front-end thread actively wakes up the waiting back-end persistence thread via a condition variable. Once awakened, the back-end thread immediately obtains the currently pending full segment (usually through a global pointer or linked list traversal) and writes all log data within that segment to disk at once. After writing, the back-end thread calls the `clear()` operation to clear the data within the segment and resets its state to "free". Simultaneously, it moves the processing pointer using the segment's `next` pointer, preparing to process the next full segment. This process achieves a complete closed loop from "segment full" to "disk flushing" to "segment reclamation." All full segments are processed sequentially until there are no more full segments to process, after which the back-end thread re-enters the waiting state.

[0049] For example, suppose a circular linked list contains three shared memory segments: segment A, segment B, and segment C, each with a capacity of 64KB. The backend thread is in a waiting state, while the frontend thread is writing logs to segment A. After writing an 8KB log entry, the used space of segment A increases from 56KB to 64KB, reaching its capacity limit, and segment A becomes full. The frontend thread immediately marks segment A as "full" and wakes up the backend thread using a condition variable, then switches to segment B to continue writing new logs. When the backend thread is awakened, it finds segment A is full, writes its 64KB log data to the disk file all at once, calls `fflush` to ensure the data is committed to the operating system, then executes `clear()` to clear segment A and reset its state to "free". Finally, it uses the `next` pointer of segment A to find segment B, discovering that segment B is not yet full (still being written to), so there are no more full segments to process, and the backend thread returns to a waiting state. Through this mechanism, the system achieves batch and efficient disk writing of log data, while the frontend and backend threads can work in parallel, significantly improving overall throughput.

[0050] Regarding the second typical scenario, if the first time difference between the current moment and the moment when log information is written to the disk exceeds the first preset time window, all log information in the circular linked list that has not been written to the disk will be written to the disk, or the log information in the shared memory segment pointed to by the current write pointer in the circular linked list will be written to the disk.

[0051] In this embodiment, the second typical scenario corresponds to a timeout triggering mechanism. Its design aims to address scenarios with low log write frequencies, preventing log data from remaining in shared memory due to the segment not being fully written for an extended period. Specifically, the system records the moment a log message is written to disk. Each time the backend persistence thread is woken up and completes a write operation, a timer is reset. When a thread wakes up due to a condition variable timeout, it calculates the time difference between the current moment and the previous write moment. If this time difference exceeds a preset first time window (e.g., 1 second), the timeout triggering condition is met.

[0052] Under the condition that the timeout triggering condition is met, the system provides two optional implementation methods to adapt to different business needs and performance considerations. The first implementation method writes all log information in the circular linked list that has not yet been written to disk. In this method, the backend thread traverses the entire circular linked list, searches for all shared memory segments in the "full" or partially written but not yet written to disk state, and writes all the data in them to disk, thus ensuring that no residual logs remain in the shared memory. The second implementation method writes only the log information in the shared memory segment currently pointed to by the write pointer to disk. The current write pointer points to the segment that the frontend business thread is currently writing to. This segment may not be full yet, but it has accumulated a certain amount of log data. By forcibly flushing it to disk, the most recently generated logs can be persisted in a timely manner with minimal I / O overhead without traversing the entire linked list. The choice between the two methods depends on the specific application scenario: if the system has extremely high data security requirements and can accept slightly higher I / O overhead, the first method can be chosen; if performance overhead is more important, and it is desirable to minimize disk operations while ensuring basic real-time performance, the second method can be chosen.

[0053] For example, suppose the first preset time window is set to 1 second, and the last disk write occurred at 10:00:00. Afterward, log writes are extremely infrequent, with only a few log entries written to segment A, which is currently pointed to by the write pointer. Segment A has ample remaining space and never becomes full. The backend thread, having no full segment to process, waits for a condition variable. When the clock reaches 10:00:01:5, the backend thread wakes up due to a timeout, calculating a time difference of 1.5 seconds, exceeding 1 second, triggering a timeout write. If the system adopts the first implementation, the backend thread will traverse all segments in the circular linked list, finding only segment A with a small amount of log data, and thus write it all to disk. If the second implementation is adopted, the backend thread directly retrieves segment A, which is currently pointed to by the write pointer, writes its data to disk, and then returns to the waiting state. Through this timeout mechanism, even without a full segment, log data will not remain in memory indefinitely, ensuring real-time performance in low-frequency scenarios.

[0054] Regarding the third typical scenario, upon receiving the thread termination signal, all log information in the circular linked list that has not been written to the disk is written to the disk.

[0055] In this embodiment, the third typical scenario corresponds to the termination mechanism when a process exits normally. Its design aims to completely write all log data in shared memory that has not yet been written to disk before the business thread actively terminates, ensuring that data is not lost due to process termination. Specifically, when the operating system or user sends a thread termination signal (such as SIGTERM) to the process, this signal is captured by a pre-registered signal handler. The signal handler follows the constraints of asynchronous signal safety, performing only minimal operations: waking up the waiting backend persistent thread by writing a byte to a dedicated pipe or event file descriptor, or by directly setting a global flag, without performing any complex I / O or memory operations in the signal context.

[0056] After the backend thread is awakened, it detects the impending process termination flag and immediately enters "emergency disk flush" mode. In this mode, the backend thread traverses the entire circular linked list, searching for all shared memory segments (i.e., all non-empty segments) that are "full" or partially written but not yet written to disk, and sequentially writes the log data in these segments to disk. For each non-empty segment, the backend thread executes the complete write process: opening or reusing the current log file, writing the data within the segment, calling fflush to ensure the data is committed to the operating system, and executing fsync to force disk write if necessary. After all non-empty segments have been processed, the backend thread exits normally, and then the main process completes final resource cleanup and terminates. This mechanism ensures that even after the process receives a termination signal, all residual log data in memory can be salvaged to disk, preserving complete context information for subsequent troubleshooting and auditing.

[0057] For example, suppose a long-running server process currently has three shared memory segments in its circular linked list: segment A is full but not yet flushed to disk, segment B is partially written (40KB used), and segment C is free. At this time, the operations team sends a SIGTERM signal to the process for planned maintenance. The signal handler is triggered, writing one byte to a dedicated pipe. The backend thread, which is waiting on a condition variable, is immediately awakened because the pipe is readable. After reading the termination flag, it begins traversing the circular linked list. First, it processes segment A, writing its 64KB log data to disk; then it processes segment B, writing 40KB of its data to disk; finally, it checks that segment C is empty and requires no processing. After all data is written to disk, the backend thread exits, and the main process completes cleanup and terminates. Through this mechanism, 104KB of log data that might have remained in memory due to process termination is completely preserved, providing a reliable basis for backtracking the system's state after maintenance.

[0058] Regarding the fourth typical scenario, if the process where the front-end business thread is located crashes, all log information in the circular linked list that has not been written to the disk will be written to the disk.

[0059] In this application embodiment, the fourth typical scenario corresponds to an emergency data recovery mechanism when a process crashes unexpectedly. Its design aims to preserve, to the greatest extent possible, the log data in shared memory that has not yet been written to disk when a business process terminates abnormally due to a segmentation fault, illegal instruction, or other unrecoverable error. Unlike termination during normal exit, a process crash cannot guarantee that the backend thread can completely execute the disk flushing operation. Therefore, this application achieves data recovery in crash scenarios by combining the persistence characteristics of shared memory with out-of-process tools.

[0060] Specifically, when a process terminates abnormally due to a crash, all threads within that process (including backend persistent threads) immediately cease execution, making it impossible to complete the normal disk flushing process. However, shared memory, as an inter-process communication resource, has a lifecycle independent of the process that created it. Even if the process crashes, the data in the shared memory segment remains intact in memory and is not automatically reclaimed by the system. This means that log information that has not yet been written to disk at the moment of the crash (including all full segments and partially written segments) still exists in the circular linked list, waiting to be salvaged.

[0061] This application implements post-crash data writing through two methods: First, the system provides an independent operation and maintenance tool, flusher, which can be manually executed by operations and maintenance personnel after a process crash. flusher appends a pre-agreed shared memory key (generated by rlog_ftok) to the main shared memory region, obtains the entry information of the circular linked list, and then traverses all shared memory segments, detecting the status and data content of each segment. For all non-empty segments (including full and partially written segments), flusher reads the data and writes it to a specified recovery file (e.g., crash_recovery.log). In this way, operations and maintenance personnel can manually salvage the last log context at the time of the crash without restarting the process. Second, this application also provides an optional automated recovery mechanism: controlled by the compiler macro ENABLE_RINGLOG_HISTORY_EXPORT, when a new process instance restarts after a crash, the system automatically detects whether there are residual non-empty segments in the shared memory. If so, the data in these segments is automatically exported to a history file, and then these segments are cleaned up for reuse, avoiding the inability of new processes to write normally due to the accumulation of residual data.

[0062] For example, suppose a process crashes during runtime due to a null pointer dereference, triggering a segmentation fault (SIGSEGV). Before the crash, there are three shared memory segments in the circular linked list: segment A is full but not yet flushed to disk (containing 64KB of logs), segment B is partially written (containing 20KB of logs), and segment C is free. After the process crashes, all threads stop, and the backend threads fail to complete the flush to disk. After the operations personnel discover the abnormal process exit, they manually run the flusher tool. The flusher attaches to the shared memory via the main shared memory key, traverses the circular linked list, detects that segments A and B are not empty, and reads a total of 84KB of log data from these two segments and writes it to the ` / var / log / crash_recovery.log` file. Subsequently, these segments can be cleaned up according to the command-line parameters. If the system has automatic export macros enabled, when a new process starts, it will automatically detect the remaining segments A and B, export them to `crash_history.log`, clean them up, and then initialize a new circular linked list normally for business use. Through this dual protection mechanism, the embodiments of this application significantly reduce the probability of log loss due to process crashes, preserving valuable critical context for root cause analysis of failures.

[0063] Further, writing the log information from at least one shared memory segment containing log information in the circular linked list to disk includes: Determine whether the first log file currently in use on the disk satisfies at least one condition in the rotation condition set; the rotation condition set includes the current system date being inconsistent with the creation date of the first log file, and the file size of the first log file being greater than or equal to a preset threshold. If the first log file satisfies at least one of the conditions in the set of rotation conditions, a new log file is created as the second log file, and the second log file is named based on the maximum global sequence number used in the metadata file in the disk and the preset naming rules. The maximum global sequence number used in the metadata file is updated. The log information to be written is written to the second log file. If the first log file does not meet any of the conditions in the set of rotation conditions, then the log information to be written is written to the first log file.

[0064] In this embodiment, when writing log information from a shared memory segment to disk, the system first performs a crucial pre-judgment: determining whether the currently used log file meets the rotation conditions. This mechanism aims to ensure that the log file size is controllable, the naming is ordered, and it is easy to retrieve and archive later through file management strategies. Specifically, before each write operation, the system checks whether the log file currently being written to on the disk (referred to as the "first log file") meets at least one condition in the rotation condition set. This rotation condition set includes two core judgment criteria: first, whether the current system date is inconsistent with the creation date of the first log file, i.e., whether it has crossed midnight; second, whether the current file size of the first log file has reached or exceeded a preset threshold (e.g., 100MB). These two conditions correspond to two typical log rotation scenarios: daily partitioning and size partitioning, respectively.

[0065] If the first log file meets any of the above conditions, the system will trigger a rotation process, creating a new log file as the "second log file". The new file's naming strictly follows a preset naming rule, typically in the format "program name.year / month / day.log.serial number". The serial number is not simply incremented, but calculated based on the "maximum used global sequence number" (seq) stored in the metadata file, specifically idx=seq%100, to ensure filenames are unique and consecutive within the same day. After generating the new filename, the system updates the global sequence number in the metadata file (e.g., incrementing seq by 1) to ensure it continues to increment in the next rotation. Subsequently, the system opens the newly created second log file and writes the currently pending log information into it. If the first log file does not meet any of the rotation conditions (i.e., the date remains unchanged and the file size does not exceed the limit), the system directly reuses the first log file, continuing to append log information, avoiding unnecessary file creation overhead.

[0066] For example, suppose a program is named "app", its log directory is " / var / log", its single segment size is 64KB, the preset file size threshold is 100MB, and the global sequence number (seq) of the current record in the metadata file is 42. The current system date is March 18, 2026, and the first log file name is / var / log / app.20260318.log.42, whose current size is 95MB, which has not yet reached 100MB. At this time, the backend thread is preparing to write a 64KB log segment. The system first checks the rotation conditions: the current system date is still March 18, 2026, which is consistent with the file creation date; the current file size of 95MB plus the 64KB to be written is approximately 95.06MB, which still does not exceed the 100MB threshold. Therefore, no rotation conditions are met, and the system directly appends the log segment to app.20260318.log.42, updating the file size to 95.06MB after writing.

[0067] Subsequently, as the log volume continues to increase, the file size eventually reaches the 100MB threshold, triggering a round-robin event on the next write operation. The system detects that the file size has reached 100MB (greater than or equal to the preset threshold), satisfying the round-robin condition. Therefore, the system reads the global sequence number 42 from the metadata file and calculates the new sequence number idx = 42%100 = 42 (note that the sequence number may repeat after modulo, but it can be distinguished by different dates). However, to maintain the incrementing logic, the actual sequence number of the new file should be calculated based on seq + 1. Assuming the system first increments seq to 43, the new file name will be / var / log / app.20260318.log.43. The system creates and opens this new file, simultaneously updating the seq in the metadata file to 43. Then, the 64KB log to be written is written to the new file. If the first log file arrives the following day (March 19, 2026), the system detects that the current date (March 19) is inconsistent with the creation date of the first log file (March 18), triggering a rotation. At this point, a new file ` / var / log / app.20260319.log.50` (since 150%100=50) is generated based on the current sequence number (let's assume it's 150), and the sequence number continues to increment. Through this mechanism, this application achieves continuous and easily searchable log file management across restarts, while avoiding operational inconvenience caused by excessively large individual files through file size threshold control.

[0068] Furthermore, the method also includes the following steps in creating a new log file as a second log file: If the number of log files on the disk exceeds the preset upper limit, the minimum global sequence number that has been used is obtained from the metadata file, and the log file corresponding to the minimum global sequence number on the disk is deleted.

[0069] In this embodiment, to effectively control long-term disk space occupation and avoid storage resource exhaustion due to the unlimited growth of log files, an automatic cleanup mechanism is introduced during the creation of new log files. Specifically, each time a new log file (i.e., the second log file) is created, the system first checks whether the total number of log files belonging to this program on the current disk exceeds a preset upper limit (e.g., retaining the most recent 100 files). This upper limit is configured during system initialization or operation to balance the retention requirements of historical logs with the limitations of disk capacity.

[0070] If the number of log files is detected to exceed a preset limit, the system will trigger a cleanup operation. The key to cleanup is determining which files should be deleted. This embodiment manages the file order using "used global sequence numbers" recorded in the metadata file: each log file name contains a sequence number generated by modulo the global sequence number (e.g., 42 in program.20260318.log.42). Since the global sequence number itself is monotonically increasing, a smaller sequence number in the filename usually corresponds to an earlier created file. To accurately locate the oldest file, the system needs to obtain the "minimum used global sequence number" from the metadata file. This minimum sequence number can be obtained by maintaining a record of used sequence numbers or by tracking the range of currently retained files each time a new file is created. After obtaining the minimum global sequence number, the system generates the corresponding filename based on this number (combining it with the date portion of the filename rules, but the dates may differ, so precise matching is required), and deletes the log file corresponding to that sequence number from the disk. Through this mechanism, the system always keeps the number of log files on the disk below the preset limit, and retains only the batch of files with the largest sequence number (i.e., the newest), achieving a first-in, first-out (FIFO) cyclic overwrite.

[0071] For example, suppose the preset maximum number of log files to retain is three. Currently, there are three log files on the disk: app.20260318.log.40 (corresponding to global sequence number 40), app.20260318.log.41 (corresponding to global sequence number 41), and app.20260318.log.42 (corresponding to global sequence number 42). When the system needs to create a new file due to a rotation, the global sequence number has increased to 43. Before creating the new file app.20260318.log.43, the system checks that the current number of files is three, reaching the limit. Therefore, the system queries the metadata file and finds that the currently used minimum global sequence number is 40. Based on this, it locates the file app.20260318.log.40 and deletes it. Then, the new file app.20260318.log.43 is created, and the number of files on the disk becomes 41, 42, and 43, maintaining the total number of three. Through this mechanism, the system always retains the three most recently generated log files, which ensures that there are enough historical logs available for backtracking and avoids the risk of unlimited disk space being occupied.

[0072] After writing the log information from at least one shared memory segment containing log information in the circular linked list to disk, the method further includes: Clear all log information written to the shared memory segment of the disk, and mark the status of the shared memory segment where the log information has been cleared as free.

[0073] In this embodiment, after the backend persistent thread successfully writes log information from a shared memory segment to disk, the system needs to reclaim that segment so that it can be reused to store new log data. This step is crucial for achieving circular buffer reuse, ensuring that a limited number of shared memory segments can be reused an unlimited number of times, thereby supporting high-throughput log writing capabilities.

[0074] Specifically, once a backend thread completes a write operation on a shared memory segment (e.g., writing the entire segment's data to a file using `fwrite` and calling `fflush` to ensure the data is committed to the operating system), the log data within that segment is safely written to disk and no longer needs to be retained in memory. At this point, the backend thread calls the `clear()` operation to logically clear the segment. This clearing operation does not actually overwrite or erase the data in memory (that would incur unnecessary CPU overhead), but rather resets the critical metadata in the segment header, primarily including setting the write pointer (such as the `used` field) within the segment to zero, indicating that there is no valid data in the current segment. Subsequently, the backend thread changes the segment's state from "Full" to "Free". This state change is crucial for the frontend business threads: once the FREE state is written to shared memory, all frontend threads can then re-identify and select this segment as a usable target segment during subsequent log writing processes.

[0075] Through this "clearing and marking free" mechanism, the system achieves the cyclical circulation of shared memory segments in a circular linked list: the front-end thread continuously writes logs to free segments, turning them into full segments; the back-end thread flushes the full segments to disk and restores them to free segments for the front-end thread to use again. This process decouples log production and consumption, enabling the efficient reuse of limited shared memory resources without the need for frequent creation and destruction of shared memory segments, significantly improving the overall performance and stability of the system.

[0076] For example, suppose there are three shared memory segments, A, B, and C, in a circular linked list. Segment A was previously filled by the front-end thread and is in a "full" state, containing 64KB of log data. After the back-end thread is awakened, it selects segment A for processing, writing the 64KB of log data to the disk file app.20260318.log.42 in one go, and calls fflush to ensure the data is written to disk. After successful writing, the back-end thread executes the clear() operation: resetting the used field in the header of segment A to 0, indicating that the data in the segment has been cleared. Then, it changes the state of segment A from "full" to "free". At this point, if the front-end thread needs to find a free segment to write new logs, and finds that segment A is in a "free" state, it will write the new log data to segment A, starting from offset 0. Through this process, segment A completes a full cycle from "free, full, flush to disk, free", continuing to provide services for subsequent log recordings.

[0077] Comprehensively, such as Figure 3As shown, when a business thread needs to record logs, it first calls the interface provided by the logging engine. The logging engine is responsible for formatting the log information into a standard string containing a timestamp and text ("formatted timestamp + text"). After formatting, the engine attempts to append this log data to the currently selected shared memory segment ("current segment cell_buffer").

[0078] During the append process, the system will determine whether there is enough remaining space in the current segment. If there is enough space, the append(log_line) operation will be executed directly to write the log data to the current segment and update the write pointer in the segment, thus ending the current write process.

[0079] If the remaining space in the current segment is insufficient to accommodate the new log, a segment switching process is triggered. First, the logging engine marks the current segment as "FULL". Then, it searches for the next segment in the circular linked list using the forward or backward pointer of the current segment. If the next segment is free, it is set as the new current segment; if all existing segments are full and the preset memory limit has not been exceeded, the system dynamically creates a new shared memory segment and inserts it into the circular linked list ("switching to the next segment / dynamically creating a new segment"). Subsequently, the logging engine re-attempts the write operation (append(log_line)) on the new segment, writing the log data into it.

[0080] Whether triggered by a segment becoming full or other circumstances, when the logging engine sets a segment to FULL, it actively wakes up the backend persistence thread via a condition variable or a signal (signal()). Once awakened, the backend thread iterates through or retrieves all currently full shared memory segments and writes their data in batches to disk files, thus achieving asynchronous log persistence. Through this process, the business thread only needs to perform lightweight memory operations, while the time-consuming disk I / O is entirely handled by the backend thread, effectively ensuring low latency and high throughput for the business thread.

[0081] like Figure 4 As shown, the backend persistence thread enters a loop processing flow after startup. The thread first waits on a condition variable and sets a timeout. When awakened by the frontend thread or automatically awakened due to timeout, the thread begins to execute the persistence operation.

[0082] First, the thread checks if there is a shared memory segment to be persisted, i.e., a segment with a "FULL" status. If there are no segments to be persisted, the thread returns to the waiting state directly, avoiding meaningless idle spinning.

[0083] If a segment needs to be persisted, the thread enters the log rotation decision phase. In this step, the thread determines whether a new log file needs to be created based on whether the current system date matches the creation date of the currently being written log file, and whether the current file size has reached a preset threshold. If the rotation conditions are met, a new file path is generated according to the naming rule of "date + sequence number," and the global sequence number in the metadata file is updated; otherwise, the current file continues to be used. If the target log file fails to open due to insufficient disk space, permission errors, or other reasons, the thread will abandon this operation and return to a waiting state to protect the data in memory from loss and to buy time for operations personnel to intervene.

[0084] After the file is successfully opened, the thread writes all the log data in the segment to be persisted to the file at once, and calls fflush to ensure that the data is committed from the process buffer to the operating system kernel, reducing the risk of data loss. After writing is complete, the thread executes the clear() operation to clear the segment and reset its state to "FREE", indicating that the segment can be reused by the front-end thread. Subsequently, the thread finds the next segment in the circular linked list through the next pointer of the current segment, moves the internal processing pointer forward, and updates the "last persisted segment" pointer recorded in the main shared memory area so that external operation and maintenance tools (such as flusher, cell_monitor) can accurately understand the current persistence progress.

[0085] After completing the above operations, the thread returns to the beginning of the loop and re-enters the state of waiting for the condition variable or timeout, awaiting the next round of processing tasks. Through this loop, the backend thread achieves efficient and reliable asynchronous log persistence, ensuring both batch writing in high-throughput scenarios and real-time performance in low-frequency scenarios through the timeout mechanism.

[0086] like Figure 5 As shown, when logs need to be written to disk, the log rotation module first checks whether the currently used log file meets the rotation conditions. The rotation conditions include two scenarios: first, the current system date is inconsistent with the creation date of the log file being written to, i.e., midnight has passed; second, the size of the current log file has reached or exceeded a preset threshold. If either condition is met, the system triggers the rotation process.

[0087] The rotation process first increments the global sequence number (seq) stored in the metadata file to ensure that each rotation obtains an incrementing unique identifier. Then, the system calculates the index value based on the incremented global sequence number using the formula idx = seq % 100. This modulo operation ensures that the generated sequence number can cycle between 0 and 99 within the same day, thus achieving filename reuse without conflicts. Next, the system generates a new log file path according to predefined naming rules, in the format {dir} / {prog}.{YYYY}{MM}{DD}.log.{idx}, where the date part is taken from the current system time, and the program name is specified during initialization.

[0088] After the new file path is generated, the system further checks whether the total number of log files belonging to this program on the current disk exceeds the preset retention limit. If it does not exceed the limit, a new file is created directly and writing is prepared; if it exceeds the limit, a cleanup operation is triggered. The cleanup operation locates and deletes the oldest log file on the disk corresponding to the smallest global sequence number recorded in the metadata file, thereby controlling the total number of files within the preset limit. After cleanup is completed, the system creates a new file and continues to perform the writing operation. If the rotation condition was not triggered during the initial check, i.e., the date has not changed and the file size has not exceeded the limit, the system directly reuses the current log file, continues to append log information, and the entire process ends.

[0089] Through this mechanism, the embodiments of this application realize automatic rotation and space reclamation of log files, which not only ensures the continuity and retrieval of file names, but also effectively controls the long-term occupation of disk space.

[0090] In summary, this embodiment of the application achieves high-speed buffering of log data and low-latency decoupling of business threads by writing the generated log information into the free shared memory segment of a pre-created circular linked list composed of at least two shared memory segments linked bidirectionally by pointers during the execution of the front-end business thread. Simultaneously, by writing the shared memory segment containing log information to disk when any one of several preset conditions is met, including a full segment in the circular linked list, timeout without writing, receiving a thread termination signal, or process crash, a multi-dimensional triggered asynchronous persistence mechanism is constructed. This method improves throughput through batch writing in normal high-concurrency scenarios, ensures real-time performance through a timeout mechanism in low-frequency scenarios, and urgently recovers data in case of abnormal exit or crash, thereby significantly improving log reliability and anomaly recovery capabilities while reducing business latency.

[0091] This application's embodiments achieve end-to-end optimization from log generation to disk persistence by constructing a complete shared-memory ring buffer log management system. This method ensures logs are written to shared memory segments in an orderly manner by introducing a current write pointer mechanism and sequential traversal judgment. When all segments are full, dynamic expansion or controlled discarding based on memory limits is performed. Simultaneously, a time window suppression strategy is combined to avoid wasting system resources due to high-frequency invalid traversal. Through a multi-condition triggered asynchronous persistence mechanism, the system can batch flush to disk when segments are full, promptly write to disk when timeouts occur, and urgently recover data when termination signals are received or processes crash, balancing high throughput, low latency, and high reliability. At the disk write level, through date and file size-based rotation judgment, globally incrementing sequence number naming, and metadata file persistence, continuous naming and automatic cleanup of log files across restarts are achieved, avoiding file overwriting and disk space overflow. Furthermore, by clearing and marking shared memory segments after writing to disk as free, the system achieves cyclic reuse of limited memory resources. In summary, the embodiments of this application significantly reduce the probability of log loss while ensuring low intrusion of business threads, improve the observability and maintenance convenience of the system, and provide an efficient, reliable and easy-to-maintain overall solution for log management in high-concurrency scenarios.

[0092] Since the electronic device described in this embodiment is an electronic device used to implement the information processing method in the embodiments of this application, those skilled in the art can understand the specific implementation methods and various variations of the electronic device in this embodiment based on the information processing method described in the embodiments of this application. Therefore, how the electronic device implements the method in the embodiments of this application will not be described in detail here. Any electronic device used by those skilled in the art to implement the information processing method in the embodiments of this application falls within the scope of protection of this application.

[0093] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0098] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A log management method, characterized in that, The method includes: During the execution of the front-end business thread, the log information generated by the front-end business thread is written into the free shared memory segment of a pre-created circular linked list. The circular linked list is composed of at least two shared memory segments linked bidirectionally by pointers. If at least one of the preset conditions is met, the log information in at least one shared memory segment containing log information in the circular linked list is written to the disk. The preset conditions include the existence of a full shared memory segment in the circular linked list, a first time difference between the current time and the time when log information was last written to the disk exceeding a first preset time window, receiving a thread termination signal, and the process in which the front-end business thread is located crashing.

2. The log management method as described in claim 1, characterized in that, Before writing the log information generated by the front-end business thread into the free shared memory segment of the pre-created circular linked list, the method further includes: Determine whether the remaining space of the current shared memory segment pointed to by the current write pointer of the front-end business thread is greater than or equal to the first space required by the first log information to be written. If the remaining space of the current shared memory segment is greater than or equal to the first space, then the current shared memory segment is determined to be a free shared memory segment; If the remaining space of the current shared memory segment is less than the first space, the current shared memory segment is determined as a full shared memory segment, and the remaining space of each shared memory segment is judged in turn according to the order of each shared memory segment in the circular linked list to see if it is greater than or equal to the first space, until a shared memory segment with a remaining space greater than or equal to the first space is found, and it is regarded as an idle shared memory segment.

3. The log management method as described in claim 2, characterized in that, After determining whether the remaining space of each shared memory segment is greater than or equal to the first space in the order of the shared memory segments in the circular linked list, the method further includes: If the remaining space of each shared memory segment in the circular linked list is less than the first space, then determine whether the memory size currently occupied by the circular linked list is greater than or equal to the preset memory limit corresponding to the circular linked list. If the memory currently occupied by the circular linked list is less than the preset memory limit, a new shared memory segment is created in the circular linked list, and the newly created shared memory segment is used as an idle shared memory segment. If the memory currently occupied by the circular linked list is greater than or equal to the preset memory limit, the first log information to be written will be discarded and the time of this discard will be recorded.

4. The log management method as described in claim 3, characterized in that, After discarding the first log message to be written and recording the time of this discard, the method further includes: If the next log message to be written is received, determine whether the second time difference between the current time and the recorded time of this discard is greater than or equal to the second preset time window; If the second time difference is less than the second preset time window, the second log information is discarded, and the time of this discard is updated. If the second time difference is greater than or equal to the second preset time window, then the remaining space of each shared memory segment in the circular linked list is determined sequentially according to the order of each shared memory segment to see if it is greater than or equal to the second space required by the second log information. If the remaining space of each shared memory segment in the circular linked list is less than the second space, the second log information is discarded and the discard time is updated; if there is a shared memory segment in the circular linked list with a remaining space greater than or equal to the second space, the shared memory segment is determined as an idle shared memory segment and the second log information is written to the idle shared memory segment.

5. The log management method as described in claim 1, characterized in that, Under the condition that at least one of the preset conditions is met, the log information in at least one shared memory segment containing log information in the circular linked list is written to the disk, including: If a shared memory segment in the circular linked list is full, the log information in the full shared memory segment is written to the disk.

6. The log management method as described in claim 1, characterized in that, Under the condition that at least one of the preset conditions is met, the log information in at least one shared memory segment containing log information in the circular linked list is written to the disk, including: If the first time difference between the current time and the time when log information was written to the disk exceeds the first preset time window, all log information in the circular linked list that has not been written to the disk will be written to the disk, or the log information in the shared memory segment pointed to by the current write pointer in the circular linked list will be written to the disk.

7. The log management method as described in claim 1, characterized in that, Under the condition that at least one of the preset conditions is met, the log information in at least one shared memory segment containing log information in the circular linked list is written to the disk, including: Upon receiving the thread termination signal, or in the event that the process containing the front-end business thread crashes, all log information in the circular linked list that has not been written to the disk will be written to the disk.

8. A log management method as described in any one of claims 1, 5, 6, and 7, characterized in that, Writing log information from at least one shared memory segment containing log information in the circular linked list to disk includes: Determine whether the first log file currently in use on the disk satisfies at least one condition in the rotation condition set; the rotation condition set includes the current system date being inconsistent with the creation date of the first log file, and the file size of the first log file being greater than or equal to a preset threshold. If the first log file satisfies at least one of the conditions in the set of rotation conditions, a new log file is created as the second log file, and the second log file is named based on the maximum global sequence number used in the metadata file in the disk and the preset naming rules. The maximum global sequence number used in the metadata file is updated. The log information to be written is written to the second log file. If the first log file does not meet any of the conditions in the set of rotation conditions, then the log information to be written is written to the first log file.

9. A log management method as described in claim 8, characterized in that, The method further includes the following steps in creating a new log file as a second log file: If the number of log files on the disk exceeds the preset upper limit, the minimum global sequence number that has been used is obtained from the metadata file, and the log file corresponding to the minimum global sequence number on the disk is deleted.

10. A log management method as described in claim 1, characterized in that, After writing the log information from at least one shared memory segment containing log information in the circular linked list to disk, the method further includes: Clear all log information written to the shared memory segment of the disk, and mark the status of the shared memory segment where the log information has been cleared as free.

Images