Log processing method, system, device and storage medium supporting multi-process security

Abstract

The application belongs to the technical field of log processing, and provides a log processing method, system, device and storage medium supporting multi-process security, wherein a multi-process security log queue with a specified capacity is created, an independent asynchronous processing thread is started and enters a loop monitoring state, a business process encapsulates a log record object containing a process ID, a name and a thread ID, and writes the log record object into the queue in a non-blocking manner; log generation and persistent writing are decoupled through queue buffering, and an output target is determined after multi-dimensional filtering of log levels, IPs and processes; writing is performed according to the target, file rotation, batch writing and network sending load control are adapted; queue backlog risks are monitored in real time, file rotation is actively triggered and abnormal self-recovery is performed. The application solves the multi-process log writing competition conflict and performance bottleneck, realizes fine filtering routing of logs, priority retention of key logs, and improves system concurrent processing, load adaptation and long-term running stability.

Description

Technical Field

[0001] This invention belongs to the field of log processing technology, specifically relating to a log processing method, system, device, and storage medium that supports multi-process security. Background Technology

[0002] As a core component of modern software architecture, logging systems record events, state changes, and error information during program execution, serving as crucial evidence for system monitoring, performance analysis, anomaly localization, and audit tracing. High-performance log processing not only requires ensuring the integrity and order of log data but also balancing system efficiency and resource utilization. Especially in multi-process concurrent environments, traditional single-threaded, lock-heavy logging methods can easily become performance bottlenecks. Current logging system designs are evolving from simple serial writing to complex systems supporting concurrent writing, structured encapsulation, and intelligent routing to meet the real-time, security, and maintainability requirements of large-scale distributed services.

[0003] A search revealed Chinese patent application CN201210562415.9, which discloses a multi-process processing method for a log system under a metadata server. This method reduces the coupling between the log system and the metadata service system and improves log processing efficiency by dividing the log system into two process spaces and performing different log processing on each space. While this method decouples the metadata service system from the log system by dividing it into two process spaces, it primarily focuses on the process partitioning structure and does not provide a complete multi-process secure write mechanism or comprehensive log processing strategy. Therefore, it cannot effectively solve the competition conflicts and performance bottlenecks during high-concurrency writes by multiple business processes. Furthermore, this existing method fails to achieve fine-grained filtering and dynamic routing selection of log levels, source IPs, and process information. It lacks adaptive adjustment capabilities when system load fluctuates or log volume surges, making it difficult to meet the practical requirements of high reliability, high scalability, and intelligent scheduling for log systems in modern distributed service environments. Summary of the Invention

[0004] The purpose of this invention is to overcome the existing defects and provide a log processing method, system, device and storage medium that supports multi-process security.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: The first objective of this invention is to provide a log processing method that supports multi-process safety, comprising the following steps: (1) Create a multi-process security log queue with a specified capacity according to the configuration parameters, start an independent asynchronous processing thread and enter the loop monitoring state; (2) The business process calls the log interface to automatically obtain the process ID, process name and thread ID of the current business process, and encapsulates these identification information with log messages, timestamps and log levels into a log recording object. The business process puts the log recording object into the multi-process security log queue in a non-blocking manner. (3) Using the multi-process security log queue as a buffer layer, log record objects put in by business processes are temporarily stored. The log generation rate and processing capacity are dynamically balanced and adjusted by buffer absorption time, decoupling efficiency coefficient and segment balancing factor to achieve decoupling of log generation and log writing operations. (4) The asynchronous processing thread retrieves the log record object from the multi-process security log queue through a blocking acquisition operation with a timeout mechanism, performs log level filtering, IP filtering and process filtering in sequence, and determines the output target of the log record object according to the filtering results and configuration rules. (5) Perform actual write operations according to the determined output target, introduce file rotation and batch write strategies, and coordinate control of network sending load coefficient to achieve dynamic balance between different output targets; (6) Monitor the length of the multi-process security log queue in each processing loop, calculate the queue backlog risk coefficient, trigger an alarm when it exceeds the preset threshold, and periodically check whether file rotation is triggered and execute a retry or recovery mechanism when a write failure or thread abnormality is detected.

[0006] Furthermore, in step (1), a multi-process security log queue with a specified capacity is created according to the configuration parameters, including: Read system-level configuration parameters, including queue capacity, log level threshold, risk coefficient threshold, and drop policy parameters; A fixed-capacity ring buffer structure is constructed based on a shared memory mapping mechanism, and a unique write sequence number generator is assigned to each business process. The queue uses a dual-index push model at its underlying level to update the write pointer and read pointer in an atomic increment manner, and introduces an atomic available space determination model to achieve ordered control when multiple business processes write concurrently.

[0007] Furthermore, in step (2), encapsulating the log recording object specifically includes: During the encapsulation phase, a globally unique identifier is generated, which is constructed by combining the process ID, thread ID, and timestamp. Calculate the importance score of the record. When the queue is full, compare the importance score of the current record with the importance score of the oldest record in the queue. If the current record has a higher importance score, replace the oldest record; otherwise, discard the current record.

[0008] Furthermore, in step (3), the multi-process security log queue is used as a buffer layer, specifically including: Log record objects are stored sequentially according to the difference between the write pointer and the read pointer, establishing a time isolation layer between production rate and processing capacity; Continuously calculate the buffer absorption time to measure the longest buffer period that the current queue can maintain without new consumption; Introducing a decoupling efficiency coefficient to characterize the degree of isolation between log generation and log writing; After each read pointer advance, the segment balance factor is updated, and the access uniformity of the circular buffer is maintained by rearranging the internal logical index mapping.

[0009] Furthermore, in step (4), log level filtering, IP filtering, and process filtering are performed sequentially, specifically including: Perform log level filtering to check if the log level has reached the output threshold, perform IP filtering to check if the log source IP is in the allowed list, and perform process filtering to selectively retain records based on the process identifier. Calculate the comprehensive filtering judgment value, and weight the log level, IP allowed indicator variable, and process allowed indicator variable using weighting coefficients. If the comprehensive filtering judgment value is greater than or equal to 1, the record is retained; otherwise, it is discarded. Calculate the routing index, and combine it with log importance score, dynamic stress index and system stability coefficient to determine the output target. When the routing index exceeds the remote output threshold, select a remote log server.

[0010] Furthermore, in step (5), the actual write operation is performed according to the determined output target, specifically including: When the target type is a local file, the current log file size and creation time are read, and the rotation trigger index is calculated by combining the preset file size threshold and time threshold. When the rotation trigger index is greater than or equal to 1, the rotation operation is triggered, a new log file is generated, and the oldest backup file is deleted according to the preset backup quantity limit. The batch write scale is determined according to the current decoupling efficiency coefficient and dynamic pressure index to maintain a dynamic balance between processing capacity and log generation rate. When the target type is console output, the process ID, process name, thread ID, log level and timestamp in the log object are concatenated into standardized text according to a unified format template, and the standard output stream or standard error stream is selected according to the log level to maintain an output ratio that is coordinated with the filtering pass rate. When the target type is a remote log server, calculate the network sending load factor to assess the network resource consumption of a single sending operation.

[0011] Furthermore, step (6) specifically includes: In each processing loop, the length of the multi-process safety log queue is monitored, the queue backlog risk coefficient is calculated, and an alarm is triggered when the threshold is exceeded. At the same time, file rotation is periodically checked and a retry or recovery mechanism is executed when a write failure or thread exception is detected.

[0012] Read the write pointer at the end of each loop. With read pointer And calculate the number of currently unprocessed logs based on the queue capacity. Based on this, a queue backlog risk coefficient is constructed, and an alarm mechanism is triggered when the queue backlog risk coefficient exceeds a preset threshold. The system periodically reads the current file size and creation time, and combines this with the rotation trigger index to make proactive predictions, calculates the rotation smoothing factor, and transforms the rotation trigger behavior into a gradual trend. Continuously monitor the number of write failures and the number of abnormal processing threads, and build a recovery priority index to achieve linkage between abnormal events and system load status.

[0013] Another object of the present invention is to provide a log processing system that supports multi-process safety, comprising: The multi-process safe queue module is used to create a log queue based on a multi-process safe mechanism. This log queue supports multiple business processes writing log records at the same time without data contention or loss, and has a capacity configuration function. When the queue is full, a non-blocking discard strategy is used to ensure the continuous operation of the system. The logging encapsulation module is used to automatically append process identification information when each log record is generated, including the process ID, process name, and thread ID of the current process, as well as record the timestamp, log level, source code file name, and line number; The asynchronous processing thread module is used to start an independent log processing thread during the system initialization phase. This log processing thread continuously monitors the log queue, retrieves log records from the queue through a blocking retrieval operation with a timeout mechanism, and passes the retrieved records to subsequent processing stages. The filtering and routing decision module is used to perform multi-dimensional filtering on log records retrieved from the queue. This includes filtering out records below a preset log level threshold, checking whether the log source IP is allowed to output based on the IP blacklist or whitelist, selectively retaining or discarding records based on process ID or name, and determining the log output target based on the filtering results and configuration rules. The output target includes local files, console standard output, or remote log servers. The persistent output module is used to actually write the filtered and routed log records to the specified output destination. It adopts thread-safe file operations when writing to the file, supports log file rotation by file size or time interval, and reduces disk I / O by batch writing multiple records. It also supports asynchronous sending when writing to the network.

[0014] Another object of the present invention is to provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the computer program to implement the log processing method supporting multi-process security provided by the first object of the present invention.

[0015] Another object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the log processing method supporting multi-process security provided by the first object of the present invention.

[0016] In combination with the above technical solutions, the beneficial effects of the present invention compared with the prior art are as follows: This invention constructs a highly reliable buffer layer between producers and consumers through a multi-process safe queue module, and introduces a dual-index advancement mechanism of write pointers and read pointers, as well as a capacity determination model, to achieve strict ordering and zero contention control during concurrent writes by multiple business processes. Combined with non-blocking write and discard policy parameters, it ensures that business processes do not experience reduced response performance due to log system blocking even under high concurrency conditions. This addresses the performance jitter problem caused by traditional synchronous log writing at the architectural level, significantly improving the overall system throughput and real-time performance.

[0017] This invention constructs a unique identifier and importance scoring model during the log recording and encapsulation stage, uniformly encoding process identification information, thread information, and time dimensions. It also establishes a risk assessment mechanism based on log level and historical alarm density, enabling logs to possess structured risk stratification capabilities before entering the queue. This not only enhances log traceability accuracy but also prioritizes the retention of critical logs in capacity-constrained scenarios, significantly improving observability and problem localization efficiency in abnormal situations.

[0018] This invention dynamically balances and adjusts the log generation rate and processing capacity by adjusting the buffer absorption time and decoupling efficiency coefficient, achieving complete decoupling between log generation and persistent writing. Even with short-term fluctuations in disk I / O or network transmission, business processes remain stable, preventing external resource jitter from affecting core business logic and improving the system's resilience and load fluctuation tolerance from a mechanism perspective.

[0019] This invention constructs a comprehensive filtering judgment value and routing selection index during the filtering and routing decision-making stages, incorporating log level, IP source, process identifier, and system load status into the decision model to achieve multi-dimensional fine-grained control. By linking with dynamic stress indicators and stability coefficients, the log output path possesses adaptive adjustment capabilities, prioritizing high-value logs to highly reliable output channels when the load increases, thereby improving the intelligence level and resource utilization efficiency of the log system.

[0020] This invention introduces a round-robin trigger index and a batch write scale calculation model in the write output stage, giving the file round-robin and batch write strategies a mathematical control basis. This effectively reduces disk I / O frequency, improves write throughput, and simultaneously controls remote output bandwidth usage through a network load factor, preventing network congestion from having a cascading impact on the system, thus improving overall log persistence efficiency and output stability.

[0021] This invention constructs a unified monitoring and recovery system through queue backlog risk coefficients, round-robin smoothing factors, and recovery priority indices. It incorporates capacity utilization ratios, rate imbalances, and abnormal events into a unified risk assessment framework, enabling automated control of alarm triggering, resource pre-allocation, and anomaly self-recovery. This mechanism ensures the log processing system remains stable under long-term operating conditions and provides quantitative data for continuous optimization and capacity planning, thereby improving system reliability and maintainability at the overall architecture level. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the structure of a log processing system supporting multi-process security provided in an embodiment of the present invention; Figure 2 This is a flowchart of a log processing method supporting multi-process safety provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the internal structure of the multi-process safe queue provided in an embodiment of the present invention; Figure 4 This is a flowchart of log recording encapsulation and importance scoring provided in an embodiment of the present invention; Figure 5 This is a flowchart of the filtering and routing decision-making process provided in an embodiment of the present invention; Figure 6 This is a flowchart of persistent output and file rotation provided in an embodiment of the present invention. Detailed Implementation

[0023] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0024] Example 1:

[0025] like Figure 1 The image shows an embodiment of a log processing system supporting multi-process safety provided by the present invention, comprising: The multi-process safe queue module is used to create a log queue based on a multi-process safe mechanism. This queue supports multiple business processes writing log records simultaneously without data contention or loss, and has a capacity configuration function. It can adopt a non-blocking discard strategy when the queue is full to ensure the continuous operation of the system. The logging encapsulation module is used to automatically attach process identification information when each log record is generated, including the process ID, process name, and thread ID of the current process, as well as record the timestamp, log level, source code file name, and line number, so as to enable accurate log tracking and process-level isolation in the future. The asynchronous processing thread module is used to start an independent log processing thread during the system initialization phase. This thread continuously monitors the log queue, retrieves log records from the queue through a blocking retrieval operation with a timeout mechanism to avoid the thread running idle due to an empty queue, and passes the retrieved records to subsequent processing stages. The filtering and routing decision module is used to perform multi-dimensional filtering on log records retrieved from the queue. This includes filtering out records below a preset log level threshold, checking whether the log source IP is allowed to output based on the IP blacklist or whitelist, selectively retaining or discarding records based on process ID or name, and determining the log output target based on the filtering results and configuration rules. The output target includes local files, console standard output, or remote log servers. The persistent output module is used to actually write the filtered and routed log records to the specified output destination. It adopts thread-safe file operations when writing to the file, supports log file rotation by file size or time interval, and can write multiple records in batches to reduce disk I / O. When writing to the network, it supports asynchronous sending to ensure the efficiency and reliability of log output.

[0026] Specifically, the key point of this embodiment lies in constructing a multi-process safe log processing system, consisting of a multi-process safe queue module, a log recording encapsulation module, an asynchronous processing thread module, a filtering and routing decision module, and a persistent output module. Each module collaborates to complete the core mechanisms and quantitative calculation models for the entire log processing process. These include a ring buffer structure based on shared memory mapping, a dual-index atomic increment capacity determination mechanism, and a non-blocking write strategy. The system also includes a structured encapsulation method for attaching process thread identifiers to log records and constructing globally unique identifiers, calculating importance scores, adaptively setting blocking waiting times for asynchronous processing threads, and a scheduling parameter calibration method for calculating initial stability coefficients. Finally, it integrates multi-dimensional filtering conditions. The system incorporates a unified monitoring, maintenance, and self-recovery system, which integrates filtering judgment values ​​and routing selection index calculation logic based on system load; unified capacity and time-based rotation trigger index; dynamically adjusted batch write scale and network transmission load coefficient control strategies; and quantitative indicators such as queue backlog risk coefficient, rotation smoothing factor, and recovery priority index for queue backlog, file rotation, and abnormal events. It also protects the collaborative operation logic that completely decouples log generation and persistent writing between modules, implements secure log writing under multi-process concurrency, fine-grained filtering routing, and priority retention of critical logs, as well as specific technical means such as thread-safe file operations, log rotation, batch writing, and asynchronous network transmission.

[0027] Example 2:

[0028] like Figure 2 As shown, this embodiment provides a log processing method that supports multi-process safety, specifically including the following steps: S1: Create a multi-process security log queue with a specified capacity according to the configuration parameters, start an independent asynchronous processing thread and enter a loop monitoring state; S2: The business process calls the log interface to automatically obtain the process ID, process name and thread ID of the current business process, and encapsulates these identification information with log messages, timestamps and log levels into a log recording object. The business process puts the log recording object into the multi-process security log queue in a non-blocking manner. S3: Using the multi-process security log queue as a buffer layer, log record objects put in by business processes are temporarily stored. The log generation rate and processing capacity are dynamically balanced and adjusted by buffer absorption time, decoupling efficiency coefficient and segment balancing factor to achieve decoupling of log generation and log writing operations. S4: The asynchronous processing thread retrieves the log record object from the multi-process security log queue through a blocking acquisition operation with a timeout mechanism, performs log level filtering, IP filtering and process filtering in sequence, and determines the output target of the log record object according to the filtering results and configuration rules; S5: Perform actual write operations based on the determined output target, introduce file rotation and batch write strategies, and coordinate control of network sending load coefficient to achieve dynamic balance between different output targets; S6: Monitor the length of the multi-process safety log queue in each processing loop, calculate the queue backlog risk coefficient, trigger an alarm when it exceeds the preset threshold, and periodically check whether file rotation is triggered and execute a retry or recovery mechanism when a write failure or thread exception is detected.

[0029] The implementation steps of this embodiment will be further explained below with reference to the component modules of the log processing system supporting multi-process security provided in Embodiment 1.

[0030] Step 1: Initialization. The multi-process security queue module creates a multi-process security log queue with a specified capacity according to the configuration parameters. At the same time, the asynchronous processing thread module starts an independent daemon thread as the log processing thread. This thread enters a loop monitoring state to wait for the arrival of queue data. In step one, the multi-process safe queue module first reads system-level configuration parameters, including queue capacity. Log level threshold Risk coefficient threshold and discard policy parameters Subsequently, a fixed-capacity circular buffer structure is constructed based on a shared memory mapping mechanism, and a unique write sequence number generator is assigned to each business process to ensure global ordering during concurrent writes by multiple producers. The queue underlying adopts a dual-index push model, with the write pointer denoted as... Read pointer is denoted as Both updates are performed atomically. To prevent write overwriting during concurrent multi-process writes, an atomic available space determination formula is introduced during the initialization phase to perform capacity verification before writing. This capacity determination value is defined as:

[0031] in Indicates the number of currently available slots. This indicates the total capacity of the configured queue. Indicates the current global write pointer position. Indicates the current global read pointer position, when Write is allowed when the value is greater than zero. When the parameter equals zero, the discard policy is triggered. The corresponding oldest record replacement rule ensures that the business process maintains non-blocking write characteristics even in high-concurrency scenarios. The purpose of this capacity determination formula is to establish a unified space constraint model during the initialization phase, enabling the multi-process safety queue module to have strict capacity boundary control capabilities during runtime, and providing basic data support for subsequent risk monitoring calculations.

[0032] After the queue structure is built, the asynchronous processing thread module immediately creates an independent daemon thread and assigns a fixed priority to it. Time slice ratio Simultaneously, a loop monitoring and scheduling model is established. To prevent thread idleness and reduce CPU utilization, the upper limit of blocking wait time is pre-calculated during the initialization phase. This time depends on the queue capacity. Compared with historical average write speed The adaptive determination is performed, and its calculation formula is as follows:

[0033] in This indicates the maximum waiting time for a single blocking acquisition. This represents the average rate at which business processes write log records per unit of time. This represents an adjustment coefficient used to control the sensitivity of the processing thread to sudden traffic surges. The purpose of this formula is to set a reasonable blocking timeout boundary for the log processing thread during the initialization phase, ensuring that the thread does not waste resources due to frequent polling, nor delay log processing due to excessive waiting time, thereby achieving a dynamic matching relationship between queue buffering capacity and processing capacity.

[0034] To further ensure stability during the high-concurrency startup phase, the initialization step also calculates the initial stability coefficient of the queue. This is used to evaluate the load balance status at the moment the processing thread starts, and its expression is:

[0035] in This indicates the processing capacity of asynchronous threads per unit of time. This indicates the log generation rate per unit time. When the value is less than one, it indicates that the processing capacity exceeds the production capacity, and the system enters a stable range. A value greater than one indicates a backlog trend, and the system automatically increases the priority of the daemon thread. And shorten This aims to enhance the consumption rate. The purpose of introducing this stability coefficient is to complete the matching assessment of production and consumption capacity during the initialization phase, reduce the risk of queue backlog from the source, and logically correspond with the risk coefficient calculation in subsequent monitoring and maintenance steps.

[0036] Through the above initialization mechanism, the multi-process safe queue module completes the establishment of capacity boundaries and the construction of a concurrent write safety model, the asynchronous processing thread module completes the adaptive setting of scheduling parameters and stability calibration, the entire system enters a loop monitoring state, and the daemon thread waits for the log recording to arrive under the blocking acquisition mechanism, thus naturally transitioning to the log generation step.

[0037] Step 2: Log generation steps, such as... Figure 4 As shown, when a business process calls the log interface during operation, the log recording encapsulation module automatically obtains the process ID, process name, and thread ID of the current process, and encapsulates these identification information along with log messages, timestamps, log levels, and other data into a log recording object. The business process then puts the log recording object into the queue managed by the multi-process safety queue module in a non-blocking manner. If the queue is full, the oldest record is discarded according to the discarding policy to ensure that the business process is not blocked. In step two, the business process triggers the log recording encapsulation module through the unified log interface. At the moment of the call, this module reads the current process ID (PID), process name (PN), and thread ID (TID) using a system call method, and simultaneously obtains the current timestamp. With log level and combined with the log message body Construct a structured logging object To ensure cross-process uniqueness and the rapid matching capability of subsequent filtering and routing decision modules, a globally unique identifier is generated during the encapsulation phase. The identifier is constructed using the following formula:

[0038] in , and A preset weight constant is used to ensure the separation of different dimension identifiers in the numerical space. PID represents the current process ID, and TID represents the thread ID. This represents the numerical representation of timestamps. The purpose of this formula is to form a stable and monotonically increasing unique sequence, ensuring the write pointer in the multi-process safe queue module is protected. A traceable mapping relationship is established between the log and the record identifier, so that the log segment from which a specific process originates can be accurately located in subsequent monitoring and maintenance steps.

[0039] After the structure encapsulation is completed, the log recording encapsulation module calculates the importance score of the records. Used in conjunction with drop policy parameters when queue capacity is limited. Collaborative work, its calculation expression is:

[0040] in , and These are the weighting coefficients. Indicates the log level as a numerical level. Indicates the current log timestamp The difference between the timestamp of the previous log entry with the same PID. This represents the historical alarm density index of the process within a unit of time. The purpose of this formula is to provide a priority ranking basis for the multi-process safety queue module, when the capacity determination formula... When the score equals zero, instead of randomly discarding the oldest record, the importance score of the oldest record is compared with the importance score of the current record. If the current record's importance score is zero, the oldest record is discarded. If the current record is larger than the oldest record, the oldest record is replaced; otherwise, the current record is discarded. This achieves a non-blocking selective discarding mechanism based on business risks, ensuring that critical logs are retained preferentially even under high load.

[0041] When the business process performs a non-blocking write, it first reads the stability coefficient calculated during the initialization phase. With the current write pointer Through atomic operations Write location The corresponding slot, and incremented. To prevent instantaneous write speed... The sharp increase caused the queue to fill up in a short time. The system updates the real-time write rate after the write is completed. The dynamic pressure index is then updated based on the following formula:

[0042] in This indicates the log write rate within the current time window. This indicates the processing capacity of asynchronous threads per unit of time. This metric is related to the initialization phase... Create an operational control group when If the log generation rate remains above one hour, it indicates that the log generation rate is higher than the processing rate. The multi-process safe queue module will increase the drop policy parameters. The corresponding replacement frequency is used to prevent business processes from being blocked due to a full queue and to maintain system real-time performance.

[0043] Through the encapsulation, scoring, and non-blocking writing mechanisms described above, the log generation steps complete the standardized construction of log objects and adaptive pressure adjustment, ensuring that writing efficiency is maintained and key records are retained first in a multi-process concurrent environment. The generated log record objects then enter the multi-process safe queue module to form a buffer layer, laying the foundation for the queue buffering and decoupling processing in the next step.

[0044] Step 3: Queue buffering step. The multi-process safe queue module acts as a buffer layer between producers and consumers, temporarily storing the log recording objects put into the business process in the queue, realizing complete decoupling of log generation and log writing operations, thereby avoiding the impact of disk I / O latency or write congestion on the execution efficiency and response speed of the business process. In step three, after completing the non-blocking write operation of the log generation step, the multi-process safe queue module will record the log object. According to the write pointer With read pointer The differences between them are stored sequentially to form logically contiguous buffer segments, thereby establishing the production rate. With processing power A time isolation layer between them. The core objective of this buffer layer is to mitigate the negative impact of disk I / O fluctuations on business processes; therefore, the buffer absorption time is continuously calculated during operation. This is used to measure the longest buffer period that the current queue can maintain without any new consumption, and its expression is:

[0045] in This indicates the number of unprocessed log records in the current queue. This represents the log write rate within the current time window. The purpose of this formula is to quantify the queue's ability to absorb sudden high-concurrency writes, especially when disk I / O latency causes... During a short period of decline, It provides acceptable time boundaries, thereby ensuring that business processes continue to execute without being affected by persistent output modules.

[0046] To further ensure the stable operation of the buffer layer, a decoupling efficiency coefficient is introduced into the system. This is used to characterize the degree of isolation between log generation and log writing, and its calculation formula is:

[0047] in Indicates the current log generation rate. This indicates the processing capacity of asynchronous threads per unit of time. This indicates the larger of the two values. The value ranges from zero to one. When it is close to one, it indicates that production and consumption are in a dynamic equilibrium, and the buffer only absorbs instantaneous fluctuations. When it decreases, it indicates that there is a sustained rate deviation. The system uses the aforementioned dynamic pressure indicators... Linkage, adjust discard strategy parameters Priority of daemon threads This restores balance. The purpose of this coefficient is to continuously evaluate whether the buffer layer has truly achieved complete decoupling during runtime, rather than relying solely on capacity. Static constraints.

[0048] To avoid cache imbalance caused by local hotspots within the queue, the module performs a check on each read pointer. Post-advancement update segment equilibrium factor Its expression is:

[0049] in Indicates the first The frequency weight of each slot being accessed per unit of time. This indicates the queue capacity. This factor reflects the uniformity of access distribution within the buffer. When the preset balance threshold is exceeded, the system rearranges the internal logical index mapping to maintain the uniformity of access to the circular buffer, thereby reducing the impact of cache thrashing on the efficiency of processing threads. The algorithm aims to ensure that the queue not only has sufficient capacity but also a stable internal access structure, providing a continuous and stable log stream for subsequent filtering and routing decision modules.

[0050] By buffer absorption time Decoupling efficiency coefficient and segment balance factor Through coordinated control, the multi-process safe queue module establishes a stable time and rate isolation layer between producers and consumers, achieving complete decoupling between log generation and persistent writing, and providing a stable data source and order guarantee for the next step of filtering and routing decision processing.

[0051] Step 4: Filtering routes, such as... Figure 5 As shown, the asynchronous processing thread module retrieves the log record object from the multi-process safety queue module through a blocking acquisition operation with a timeout mechanism, and then passes the record object to the filtering and routing decision module. The filtering and routing decision module sequentially performs log level filtering to check whether the log level has reached the output threshold, IP filtering to check whether the log source IP is in the allowed list, and process filtering to selectively retain records based on the process identifier. Based on the filtering results and pre-configured routing rules, it determines whether the record should be output to a local file, the console, or a remote log server. In step four, the asynchronous processing thread module calculates the blocking wait time based on the initialization phase. Execute a blocking acquire operation with a timeout mechanism when reading the pointer. Less than write pointer Retrieve the log record object from the corresponding slot at time. and increase synchronously. To maintain the balance factor with the buffer zone A consistent, sequential pace was maintained. (Achievements) The system then immediately proceeds to the filtering and routing decision module, which first calculates the comprehensive filtering decision value. This is used to uniformly express the combined results of log-level filtering, IP filtering, and process filtering. Its expression is:

[0052] in Indicates the log level in the log record. This indicates the preset log level threshold. This is an IP allow indicator variable; it takes a value of one if the source IP is in the allow list, and a value of zero otherwise. This is a process permission indicator variable. It takes a value of one when the PID or PN meets the process filtering rules, and a value of zero otherwise. , and These are the weighting coefficients. The purpose of this formula is to transform multi-dimensional filtering logic into a single continuous decision space, when... If the value is greater than or equal to 1, the record is retained; otherwise, it is discarded. This avoids the overhead of multiple nested conditions and improves processing capacity. Effective utilization rate.

[0053] when Once the output conditions are met, the module calculates the routing index. This is used to determine the output target type, and its expression is:

[0054] in This represents the importance score calculated during the log generation process. This indicates a dynamic pressure indicator. This represents the stability coefficient during the initialization phase. , and is the weighting parameter. The purpose of this formula is to make output path decisions by comprehensively considering the log risk level and the current system load status. When the output file size is below the preset output threshold, select a local file. When in the middle range, select console output. When the remote output threshold is exceeded, a remote log server is selected, thus prioritizing the entry of highly important logs into a highly reliable storage channel when the system load increases.

[0055] To ensure that the filtration process does not compromise the aforementioned decoupling efficiency coefficient The established rate balance allows the module to update the filter pass rate after each batch of processing. Its expression is:

[0056] in This indicates the number of logs that passed the filter within the current time window. Indicates the total number of logs processed. When A decrease indicates increased filtering intensity and reduced processing thread load, which helps with recovery. and The balance between them, when Rising and When the level remains consistently high, the system increases... Alternatively, the IP filtering list can be tightened to prevent queue backlog. The purpose of this pass rate calculation is to link filtering with the overall system buffer stability, forming a closed-loop control structure.

[0057] After comprehensive filtering and routing, the log recording object is marked with a specific output target and enters the persistent output module with the target type identifier, thus naturally transitioning to the write output step.

[0058] Step 5: Write the output steps, such as... Figure 6 As shown, the persistent output module performs the actual write operation according to the target type determined by the filtering and routing decision module. For file output, a thread-safe file append writing method is adopted. Before writing, the current file size or creation time is checked to determine whether the file rotation condition is triggered. If triggered, the rotation operation is performed to generate a new log file and retain a specified number of backup files. At the same time, it supports merging multiple log records into batch writing to reduce disk I / O. For console output, it prints to standard output or standard error stream according to the specified format. For network output, it sends log records to a remote log server through asynchronous sockets. In step five, the persistent output module receives a log recording object carrying a target type identifier. Based on the routing index The determined output path executes the corresponding write strategy. When the target type is a local file, the module first reads the current log file size. With creation time And combined with a preset file size threshold With time threshold Calculate the rotation trigger index Its expression is:

[0059] in Indicates the current timestamp. This indicates that the current file has already occupied space. Indicates the maximum allowed file size. Indicates the current file creation time. Indicates the maximum allowed duration. When When the value is greater than or equal to 1, a rotation operation is triggered, a new log file is generated, and the oldest backup file is deleted according to the preset backup limit, thereby ensuring stable and controllable disk space. The purpose of this formula is to unify the two dimensions of capacity and time into a single judgment framework, avoiding the abnormal expansion of log files or excessively frequent rotations caused by relying on a single condition.

[0060] Before performing the actual write operation, the module determines the current decoupling efficiency coefficient. With dynamic pressure indicators Determine the batch write size This is used to reduce disk I / O operations, and its calculation formula is:

[0061] in This indicates the capacity of the multi-process safe queue module. This represents the decoupling efficiency coefficient. Represents a dynamic pressure indicator, symbol This indicates a floor function. The purpose of this formula is to increase the batch write size to improve throughput when the system is stable, and to automatically reduce the batch size to reduce write latency when the pressure increases, thereby maintaining processing capacity. With log generation rate A dynamic balance is maintained between them. Batch writes are performed using a thread-safe append-only method, and processing capacity is updated after the write is complete. The real-time estimates provide a basis for subsequent stress assessments.

[0062] When the target type is console output, the module will output according to the unified format template. PID, PN, TID, With timestamp Concatenate into standardized text and categorize according to log level. Choose either the standard output stream or the standard error stream to maintain the filter pass rate. A coordinated output ratio. When the target type is a remote log server, the module calculates the network transmission load factor. Its expression is:

[0063] in Indicates the current batch record quantity. This represents the average size in bytes per log entry, and BW represents the currently available network bandwidth. This coefficient is used to evaluate the network resource consumption of a single transmission operation. As time approaches, the system automatically sends data in batches to avoid network congestion, thereby ensuring stable reception by the remote log server.

[0064] Trigger index through rotation Batch write scale and network transmission load factor Through collaborative control, the persistent output module achieves a dynamic balance between efficiency and stability across different output targets, while continuously providing file status and processing capabilities to the monitoring and maintenance steps. Sending load data lays the data foundation for the next step of risk assessment and anomaly recovery mechanism.

[0065] Step Six: Monitoring and Maintenance. In each processing loop, the asynchronous processing thread module monitors the current queue length of the multi-process safe queue module, calculates the queue backlog risk coefficient, and compares it with a preset threshold. When the risk coefficient exceeds 80%, an alarm is triggered to notify the system administrator. At the same time, the module periodically checks the size and creation time of the log files managed by the persistent output module. If the file rotation condition is met, the module actively triggers the rotation operation. When a write failure or processing thread abnormality is detected, a retry or recovery mechanism is executed to ensure the long-term stable operation of the log processing system.

[0066] In step six, the asynchronous processing thread module reads the write pointer at the end of each loop. With read pointer And combined with queue capacity Calculate the number of currently unprocessed logs. Based on this, a queue backlog risk coefficient is constructed. It is used to characterize the system load state, and its expression is:

[0067] in This indicates the current number of items in the queue. Indicates the capacity of the multi-process safe queue module. Indicates the current log generation rate. This represents the processing capacity per unit time of asynchronous processing threads. The purpose of this formula is to unify the capacity utilization ratio and the degree of rate imbalance into a single risk indicator. An alarm mechanism is triggered when the value is greater than or equal to 0.8, and the current value is recorded. and This value is used by system administrators to analyze load sources. This risk coefficient reflects not only the static backlog level but also the dynamic difference between production and consumption capacity, thus avoiding false alarms caused by relying solely on queue length as a judgment criterion.

[0068] In terms of file maintenance, the module periodically reads the current file size. With creation time And combined with rotation trigger index Actively predict and avoid triggering rotation during peak write periods, which could disrupt instantaneous processing capacity. The system calculates the rotation smoothing factor as the frequency decreases. Its expression is:

[0069] in For smoothing coefficients, This is the rotation trigger index defined in the write-out step. This factor translates rotation trigger behavior into a gradual trend, when... Approaching a time, As the process gradually escalates to level one, the system pre-allocates new file handles and buffer resources, enabling a rapid switchover when the actual rotation conditions are met, thus avoiding impact on the scale of batch writes. With network transmission load factor Stability.

[0070] In terms of anomaly recovery, the module continuously monitors the number of write failures. Exception count of handling threads And construct a recovery priority index Its expression is:

[0071] in , and For weight parameters, This indicates the number of write failures per unit of time. This indicates the exception count for the handling thread. This represents the queue backlog risk coefficient. When... When the preset recovery threshold is exceeded, the system immediately executes a tiered recovery strategy: first, it resets the connection state of the persistent output module, then restarts the asynchronous processing thread module, and recalibrates the processing capacity. With log generation rate Real-time estimates to recover To reach a stable range. The purpose of this recovery index is to link abnormal events with system load status, preventing a single anomaly from being amplified into a system-wide failure.

[0072] Risk coefficient based on queue backlog Rotational smoothing factor and recovery priority index The collaborative control and monitoring and maintenance steps enable unified governance of capacity, rate and anomalies, ensuring the long-term stable operation of the multi-process safety queue module, filtering and routing decision module and persistent output module, thereby completing the log processing closed loop and providing the operational data foundation for continuous system optimization.

[0073] Example 3:

[0074] This embodiment provides an electronic device, including a processor and a memory storing a computer program. When the processor executes the computer program, it implements the log processing method supporting multi-process security provided by the first objective of this invention.

[0075] Example 4:

[0076] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the log processing method supporting multi-process security provided by the first objective of this invention.

[0077] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in the present invention, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0078] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0079] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A log processing method supporting multi-process safety, characterized in that, The method includes the following steps: (1) Create a multi-process security log queue with a specified capacity according to the configuration parameters, start an independent asynchronous processing thread and enter the loop monitoring state; (2) The business process calls the log interface to automatically obtain the process ID, process name and thread ID of the current business process, and encapsulates these identification information with log messages, timestamps and log levels into a log recording object. The business process puts the log recording object into the multi-process security log queue in a non-blocking manner. (3) Using the multi-process security log queue as a buffer layer, log record objects put in by business processes are temporarily stored. The log generation rate and processing capacity are dynamically balanced and adjusted by buffer absorption time, decoupling efficiency coefficient and segment balancing factor to achieve decoupling of log generation and log writing operations. (4) The asynchronous processing thread retrieves the log record object from the multi-process security log queue through a blocking acquisition operation with a timeout mechanism, performs log level filtering, IP filtering and process filtering in sequence, and determines the output target of the log record object according to the filtering results and configuration rules. (5) Perform actual write operations according to the determined output target, introduce file rotation and batch write strategies, and coordinate control of network sending load coefficient to achieve dynamic balance between different output targets; (6) Monitor the length of the multi-process security log queue in each processing loop, calculate the queue backlog risk coefficient, trigger an alarm when it exceeds the preset threshold, and periodically check whether file rotation is triggered and execute a retry or recovery mechanism when a write failure or thread abnormality is detected.

2. The log processing method supporting multi-process safety according to claim 1, characterized in that, In step (1), a multi-process security log queue with a specified capacity is created according to the configuration parameters, including: Read system-level configuration parameters, including queue capacity, log level threshold, risk coefficient threshold, and drop policy parameters; A fixed-capacity ring buffer structure is constructed based on a shared memory mapping mechanism, and a unique write sequence number generator is assigned to each business process. The queue uses a dual-index push model at its underlying level to update the write pointer and read pointer in an atomic increment manner, and introduces an atomic available space determination model to achieve ordered control when multiple business processes write concurrently.

3. The log processing method supporting multi-process safety according to claim 1, characterized in that, In step (2), encapsulating the log recording object specifically includes: During the encapsulation phase, a globally unique identifier is generated, which is constructed by combining the process ID, thread ID, and timestamp. Calculate the importance score of the record. When the queue is full, compare the importance score of the current record with the importance score of the oldest record in the queue. If the current record has a higher importance score, replace the oldest record; otherwise, discard the current record.

4. The log processing method supporting multi-process safety according to claim 1, characterized in that, In step (3), the multi-process security log queue is used as a buffer layer, specifically including: Log record objects are stored sequentially according to the difference between the write pointer and the read pointer, establishing a time isolation layer between production rate and processing capacity; Continuously calculate the buffer absorption time to measure the longest buffer period that the current queue can maintain without new consumption; Introduce a decoupling efficiency coefficient to characterize the degree of isolation between log generation and log writing; After each read pointer advance, the segment balance factor is updated, and the access uniformity of the circular buffer is maintained by rearranging the internal logical index mapping.

5. The log processing method supporting multi-process safety according to claim 1, characterized in that, In step (4), log level filtering, IP filtering, and process filtering are performed sequentially, specifically including: Perform log level filtering to check if the log level has reached the output threshold, perform IP filtering to check if the log source IP is in the allowed list, and perform process filtering to selectively retain records based on the process identifier. Calculate the comprehensive filtering judgment value, and weight the log level, IP allowed indicator variable, and process allowed indicator variable using weighting coefficients. If the comprehensive filtering judgment value is greater than or equal to 1, the record is retained; otherwise, it is discarded. Calculate the routing index, and combine it with log importance score, dynamic stress index and system stability coefficient to determine the output target. When the routing index exceeds the remote output threshold, select a remote log server.

6. The log processing method supporting multi-process safety according to claim 1, characterized in that, In step (5), the actual write operation is performed according to the determined output target, specifically including: When the target type is a local file, the current log file size and creation time are read, and the rotation trigger index is calculated by combining the preset file size threshold and time threshold. When the rotation trigger index is greater than or equal to 1, the rotation operation is triggered, a new log file is generated, and the oldest backup file is deleted according to the preset backup quantity limit. The batch write scale is determined according to the current decoupling efficiency coefficient and dynamic pressure index to maintain a dynamic balance between processing capacity and log generation rate. When the target type is console output, the process ID, process name, thread ID, log level and timestamp in the log object are concatenated into standardized text according to a unified format template, and the standard output stream or standard error stream is selected according to the log level to maintain an output ratio that is coordinated with the filtering pass rate. When the target type is a remote log server, calculate the network sending load factor to assess the network resource consumption of a single sending operation.

7. The log processing method supporting multi-process safety according to claim 1, characterized in that, Step (6) specifically includes: In each processing loop, monitor the length of the multi-process safety log queue, calculate the queue backlog risk coefficient, trigger an alarm when it exceeds the preset threshold, and periodically check whether file rotation is triggered and execute a retry or recovery mechanism when a write failure or thread exception is detected. Read the write pointer at the end of each loop. With read pointer And calculate the number of currently unprocessed logs based on the queue capacity. Based on this, a queue backlog risk coefficient is constructed, and an alarm mechanism is triggered when the queue backlog risk coefficient exceeds a preset threshold. The system periodically reads the current file size and creation time, and combines this with the rotation trigger index to make proactive predictions, calculates the rotation smoothing factor, and transforms the rotation trigger behavior into a gradual trend. Continuously monitor the number of write failures and the number of abnormal processing threads, and build a recovery priority index to achieve linkage between abnormal events and system load status.

8. A log processing system supporting multi-process safety, characterized in that, The system includes: The multi-process safe queue module is used to create a log queue based on a multi-process safe mechanism. This log queue supports multiple business processes writing log records at the same time without data contention or loss, and has a capacity configuration function. When the queue is full, a non-blocking discard strategy is used to ensure the continuous operation of the system. The logging encapsulation module is used to automatically append process identification information when each log record is generated, including the process ID, process name, and thread ID of the current process, as well as record the timestamp, log level, source code file name, and line number; The asynchronous processing thread module is used to start an independent log processing thread during the system initialization phase. This log processing thread continuously monitors the log queue, retrieves log records from the queue through a blocking retrieval operation with a timeout mechanism, and passes the retrieved records to subsequent processing stages. The filtering and routing decision module is used to perform multi-dimensional filtering on log records retrieved from the queue. This includes filtering out records below a preset log level threshold, checking whether the log source IP is allowed to output based on the IP blacklist or whitelist, selectively retaining or discarding records based on process ID or name, and determining the log output target based on the filtering results and configuration rules. The output target includes local files, console standard output, or remote log servers. The persistent output module is used to actually write the filtered and routed log records to the specified output destination. It adopts thread-safe file operations when writing to the file, supports log file rotation by file size or time interval, and reduces disk I / O by batch writing multiple records. It also supports asynchronous sending when writing to the network.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the log processing method supporting multi-process security as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the log processing method supporting multi-process safety as described in any one of claims 1 to 7.

Images