Log processing method and device, electronic equipment and readable storage medium

Abstract

Embodiments of the present application provide a log processing method and device, electronic equipment and storage medium, wherein before a target thread executes a log write operation, if the split information of the log file path bound by the target thread meets the log split condition, the target thread splits the original log file pointed to by the log file path to obtain a target split file; the target thread is any thread of at least two threads, different threads bind different log file paths and log queues; based on the target thread, the log to be written in the log queue bound by the target thread is obtained; and the log write operation is performed on the log to be written to the target split file. In this way, since different threads bind different log file paths and log queues, thread conflicts can be avoided through independent operation channels of threads, queues and files, log processing reliability is improved, and data integrity is ensured by ensuring that threads process logs in their corresponding queues in order.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a log processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the widespread application of basic service frameworks in complex business scenarios, massive amounts of information and error logs generated during task execution are written to log files. These logs, as the core record carriers of system operation status, can become the key basis for locating the cause of anomalies and tracing the root cause of problems.

[0003] In related technologies, when multiple threads concurrently trigger log write events, if at least two threads simultaneously detect the need for log splitting operations, it may cause at least two threads to perform log splitting and log writing operations on the same log file at the same time, resulting in multi-threaded conflicts and causing problems such as log data loss, log writing errors, and loss of log writing objects, which cannot guarantee data integrity and log processing reliability. Summary of the Invention

[0004] The purpose of this invention is to provide a log processing method, apparatus, electronic device, and storage medium to improve the reliability of log processing. The specific technical solution is as follows: In a first aspect of this invention, a log processing method is provided, the method comprising: Before the target thread performs the log writing operation, if the splitting information of the log file path bound to the target thread meets the log splitting conditions, then the target thread splits the original log file pointed to by the log file path to obtain the target split file; the target thread is any one of at least two threads, and different threads are bound to different log file paths and log queues; Based on the target thread, retrieve the logs to be written from the log queue bound to the target thread; Perform a log writing operation on the log to be written to the target split file.

[0005] In a second aspect of the invention, a log processing apparatus is also provided, the apparatus comprising: The first segmentation module is used to segment the original log file pointed to by the log file path before the target thread performs the log writing operation. If the segmentation information of the log file path bound to the target thread meets the log segmentation conditions, the target thread will segment the original log file to obtain the target segmented file. The target thread is any one of at least two threads, and different threads are bound to different log file paths and log queues. The first acquisition module is used to acquire logs to be written from the log queue bound to the target thread, based on the target thread. The first writing module is used to perform a log writing operation on the log to be written to the target split file.

[0006] In another aspect of the present invention, a computer-readable storage medium is also provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform any of the log processing methods described above.

[0007] In another aspect of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the log processing methods described above.

[0008] The log processing method, apparatus, electronic device, and storage medium provided in this invention, before the target thread performs a log writing operation, if the segmentation information of the log file path bound to the target thread meets the log segmentation conditions, then the target thread segments the original log file pointed to by the log file path to obtain a target segmented file. The target thread can be any of at least two threads, with different threads bound to different log file paths and log queues. Based on the target thread, logs to be written in the log queue bound to the target thread are obtained; and a log writing operation is performed on the logs to be written to the target segmented file. Thus, since different threads are bound to different log file paths, the original log files of different threads are different, and correspondingly, the target segmented files obtained by each thread segmenting the original log file pointed to by its respective log file path are different. Therefore, in multi-threaded log writing and segmentation scenarios, log writing conflicts can be avoided, ensuring the integrity of log data and the reliability of log processing. Simultaneously, since each thread is bound to a different log queue, each thread can execute in parallel, writing the logs to be written in the log queue to its respective target segmented file. This avoids log writing conflicts while ensuring the parallelism of the solution, thereby ensuring log writing efficiency. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0010] Figure 1 This is a flowchart of the steps of a log processing method provided in an embodiment of the present invention; Figure 2 This is an initialization flowchart provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of the initialization process of a first log processor provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a locking process provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a process for reading the log segmentation time point and writing the next log segmentation time point provided by an embodiment of the present invention; Figure 6 This is a schematic diagram of the processing flow of a log processor provided in an embodiment of the present invention; Figure 7 This is a structural diagram of a log processing device provided in an embodiment of the present invention; Figure 8 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0012] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0013] The log processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0014] Suppose there are two threads, a and b. In related technologies, when multiple threads simultaneously write logs to the log file main.log, and the log file main.log has reached its log splitting time, the next log file to be created after the split will be named main.log.1. In this scenario, because both threads a and b will perform the splitting operation on the log file main.log, that is, multiple threads will write logs and split logs on the same original log file pointed to by the same log file path, which may lead to problems such as log data loss, log writing errors, and loss of log writing objects. Specifically: When the log splitting conditions are met, if thread A triggers the log splitting event first, it will rename main.log to main.log.1 and then create a new main.log. However, before thread A's splitting operation is finished, thread B also begins its log splitting. Thread B detects that a main.log.1 file already exists in the current path and deletes it. As a result, all log data recorded in main.log (now renamed main.log.1) before the split will be lost.

[0015] Continuing the above process, after deleting main.log.1, thread b also renames main.log (at this point, the new main.log has been created by a, but thread b may not be aware of this and will still process it according to its own logic) to main.log.1. Now, thread a continues to write logs to what it considers the current log file (i.e., the renamed main.log.1), while thread b writes logs to its newly created (or writable) main.log. This leads to disordered log writing, with log data from different threads mixed in incorrect files.

[0016] In a more extreme scenario, thread A renames main.log to main.log.1 but hasn't created a new main.log file. At this point, thread B triggers a log splitting event, deleting main.log.1. Since thread A's write handle still points to the deleted main.log.1 file, thread A permanently loses its log writing object and thus permanently loses its ability to write logs. To address this, this invention provides a log processing method to solve the aforementioned technical problem.

[0017] Figure 1 This is a flowchart of the steps of a log processing method provided in an embodiment of the present invention, as follows: Figure 1 As shown, the method may include: Step 101: Before the target thread performs the log writing operation, if the splitting information of the log file path bound to the target thread meets the log splitting conditions, the target thread splits the original log file pointed to by the log file path to obtain the target split file; the target thread is any one of at least two threads, and different threads are bound to different log file paths and log queues.

[0018] In this embodiment of the invention, at least two log queues are pre-created according to the log type. Simultaneously, the same number of first log processors are set according to the number of log queues. Each log queue corresponds to one first log processor, and each first log processor creates an independent thread. Each thread is bound to a different log queue; that is, each thread receives logs from a corresponding log queue and performs a log write operation, writing the logs to the log file corresponding to that thread. In other words, each thread created by a log processor uniquely corresponds to a log queue and the file path of the log file, forming a one-to-one mapping relationship of "thread-queue-file". Furthermore, the thread operations of each thread are independent and do not interfere with each other. The first log processor can be a timed rotating file handler (e.g., NewTimed Rotating File Handler), used to read logs from the bound log queue and write them to the log file pointed to by the corresponding file path by starting an independent thread. The thread operations can include log write operations and log splitting operations.

[0019] In a concurrent environment consisting of at least two threads, before any target thread performs a log write operation, it can first check whether the partitioning information of the log file path corresponding to the target thread meets the preset log partitioning conditions. If the log partitioning conditions are met, the target thread atomically completes the partitioning operation of the original log file, generating the target partitioned file. The target thread can be the thread preparing to perform the log write operation. The log file path can be determined by a pre-defined target file path object corresponding to the target thread. The log file path object can be a composite object containing metadata such as file path, file handle, and write position pointer. Partitioning information can include file size, the number of logs contained in the file, and a preset partitioning time. Log partitioning conditions can include the log file size reaching a preset size threshold, the number of log records reaching a preset value, or the preset partitioning time reaching or exceeding the current time.

[0020] For example, the log splitting method can be as follows: First, check if a file corresponding to the log name to be created exists in the current log file path. If a file corresponding to the log name to be created exists, the target thread will delete the file and rename the current original log file. Then, the target thread will create a new log file and set the name of the new log file to the original name of the original log file. This new log file will serve as the target splitting file for subsequent log entries. All subsequent log records will be written to this new target splitting file until the log splitting condition is triggered again, at which point the above splitting process will be repeated on the target splitting file.

[0021] Step 102: Based on the target thread, obtain the logs to be written in the log queue bound to the target thread.

[0022] Step 103: Perform a log writing operation on the log to be written to the target split file.

[0023] In this embodiment of the invention, after log splitting is completed, the target split file can be used as the new write file path for subsequent threads. Since the original log file is renamed (i.e., the physical file is changed) during log splitting, but the target split file is renamed to the original name of the original log file, the thread's log write operation still uses the original log file path, which automatically points to the newly created empty file. No configuration modification is required for subsequent threads to write logs to the target split file based on the corresponding log file path. Then, the target thread can retrieve logs to be written from its bound log queue in a first-in, first-out (FIFO) order, and write them to the newly split target split file corresponding to the target thread according to a preset format and rules. Since each thread has its own independent log queue, the target thread will sequentially retrieve logs from its bound log queue and write them to the corresponding log file as logs to be written. During this process, there will be no conflict with other threads, and each thread performs only one log write operation at a time.

[0024] In summary, in this embodiment of the invention, before the target thread performs the log writing operation, if the partitioning information of the log file path bound to the target thread meets the log partitioning conditions, the target thread partitions the original log file pointed to by the log file path to obtain the target partitioned file. The target thread can be any of at least two threads, each bound to a different log file path and log queue. Based on the target thread, logs to be written in the log queue bound to the target thread are obtained. The log writing operation is then performed on the logs to be written to the target partitioned file. Thus, because different threads are bound to different log file paths, the original log files of different threads are different. Consequently, the target partitioned files obtained by each thread partitioning the original log file pointed to by its respective log file path are different. Therefore, in multi-threaded log writing and partitioning scenarios, log writing conflicts can be avoided, ensuring the integrity of log data and the reliability of log processing. Simultaneously, because each thread is bound to a different log queue, each thread can execute in parallel, writing the logs to be written in the log queue to its respective target partitioned file. This avoids log writing conflicts while ensuring the parallelism of the solution, thereby ensuring log writing efficiency.

[0025] Optionally, embodiments of the present invention may further include the following steps: Step 201: If the log file path segmentation information corresponding to the target thread does not meet the log segmentation conditions, obtain the log to be written from the log queue bound to the target thread based on the target thread.

[0026] In this embodiment of the invention, if it is determined that the log file path corresponding to the target thread does not meet the log splitting conditions, such as the log file size not reaching a preset size threshold, the number of log records not reaching a preset value, or the current time not reaching a preset splitting time, then the log writing process in the non-splitting scenario is entered. Based on the target thread, logs to be written are sequentially retrieved from the log queue bound to the target thread.

[0027] Step 202: Perform a log writing operation on the log to be written to the original log file.

[0028] In this embodiment of the invention, the writing target, i.e. the original log file, is obtained from the log file path object bound to the target thread, and the log to be written is written to the original log file.

[0029] In this embodiment of the invention, when the partitioning information of the log file path corresponding to the target thread does not meet the log partitioning conditions, logs to be written are obtained from the log queue bound to the target thread based on the target thread; and log writing operations are performed on the logs to be written to the original log file. This enables the target thread to obtain logs from the unique log queue it is bound to and write them to the original log file bound to the target thread when the partitioning conditions are not met, avoiding multi-threaded contention for shared resources and ensuring the order and integrity of logs.

[0030] Optionally, embodiments of the present invention may further include the following steps: Step 301: For any log type, create a log queue corresponding to that log type.

[0031] In this embodiment of the invention, an independent First-In-First-Out (FIFO) queue structure is created for each log type by enumerating log types or based on a set of log types defined in a configuration file. This structure serves as the log queue corresponding to each log type. Log files corresponding to any log type are distributed to the log queue corresponding to that log type.

[0032] Step 302: For any of the log queues, create the thread corresponding to the log queue and the original log file corresponding to the thread.

[0033] In this embodiment of the invention, an independent thread is created for each log queue, and the thread is bound to the log queue in a one-to-one manner. Simultaneously, a unique original log file is generated based on the log type. The filename of the original log file can be generated using predefined rules, such as the format "log type_timestamp.log".

[0034] Step 303: Bind the log file path corresponding to the original log file to the thread; different log file paths point to different filenames of the original log files.

[0035] In this embodiment of the invention, log file paths are associated with thread instances. For example, the association between log file paths and threads can be achieved through thread configuration objects. After the log file paths are bound, all write operations of a thread locate the file to be written to through its corresponding log file path, eliminating the need for dynamic path resolution each time a write operation is performed. Different threads are bound to paths that point to different original log file names. This ensures that when each thread performs log splitting on the log file of its bound path, its operation only applies to its own bound path, unaffecting other threads and preventing log splitting errors due to filename confusion.

[0036] In this embodiment of the invention, by constructing a one-to-one mapping relationship between "thread-queue-file", it is ensured that each log type is buffered by an independent queue, written by a dedicated thread, and located by a unique path, thereby achieving thread isolation in log processing. Furthermore, different log file paths point to different filenames of the original log files, which can avoid log corruption or loss caused by filename conflicts during multi-threaded concurrent splitting, thus improving the reliability and maintainability of log management in high-concurrency scenarios.

[0037] Optionally, embodiments of the present invention may further include the following steps: Step 401: When a log write event is triggered, based on the mapping relationship between log type and log queue, determine the log queue that matches the log type of the log indicated by the log write event, and use it as the log queue to be distributed.

[0038] Step 402: Distribute the log written by the event to the log queue to be distributed.

[0039] In this embodiment of the invention, when the system generates logs, a log write event is triggered. In response to the log write event, a second log processor performs formatted preprocessing on the logs indicated by the log write event and then distributes them. The second log processor can be a multi-queue log processor (MultiQueueHandler) used to manage different log queues and is responsible for log classification and routing. After receiving logs, it can distribute them to different log queues. Formatted preprocessing refers to converting the received logs into a uniform format.

[0040] Log types are pre-determined for different logs. These log types can be distinguished based on log levels, ranging from low to high: DEBUG, INFO, WARN, ERROR, and FATAL. This results in five different log types: DEBUG, INFO, WARN, ERROR, and FATAL. It is understood that log types can be set according to actual log storage needs, and this embodiment of the invention does not impose such limitations. For at least two log types, log queues corresponding to each log type are created, such as an error log queue and an information log queue. Based on the mapping relationship between log types and log queues, a log queue matching the log type indicated by the log write event is determined as the log queue to be distributed. For example, after creating the log queue, the mapping relationship between the log queue and the log type can be recorded to obtain a queue mapping table. Then, based on the log type corresponding to the log indicated by the log write event, the second log processor queries the queue mapping table to obtain the log queue to be distributed that matches the log type, and distributes the log to the log queue matching the log type.

[0041] In this embodiment of the invention, when a log write event is triggered, the log is routed to the corresponding log queue to be distributed according to its type attribute, thereby achieving classified storage and distribution of logs. This improves log processing efficiency and system stability in high-concurrency scenarios while ensuring log integrity.

[0042] Optionally, step 302 may include the following steps: Step 501: For any of the log queues, create a first log processor corresponding to the log queue.

[0043] Step 502: Create the thread corresponding to the log queue through the first log processor.

[0044] In this embodiment of the invention, after creating at least two log queues, a first log processor is created for each log queue, and a thread is created through the first log processor. By creating a corresponding thread for each log queue, different threads can process different log queues in parallel without interfering with each other, thus speeding up log processing.

[0045] Accordingly, step 402 may include the following steps: Step 503: Distribute the log written by the second log processor to the log queue to be distributed.

[0046] In this embodiment of the invention, in the actual log processing flow, log write events may be triggered by different sources, such as different business modules or different user operations. Logs generated from different sources need to be distributed to the corresponding log queues to be distributed by the second log processor. Therefore, after determining the log queues to be distributed according to the log type, the log indicated by the log write event is distributed to the corresponding log queues to be distributed by the second log processor.

[0047] In this embodiment of the invention, by creating a first log processor corresponding to the log queue and creating threads through the first log processor, each thread can operate in parallel under this architecture, performing independent and non-interfering processing on different log queues, thereby improving the speed and efficiency of log writing. The second log processor distributes logs to the queue to be distributed according to the log type, realizing the ordered distribution of logs under the multi-log queue architecture.

[0048] Optionally, step 101 may include the following steps: Step 601: The target thread changes the original file name of the original log file to a first file name; the first file name is obtained by adding a first identifier to the original file name.

[0049] In this embodiment of the invention, during the process of the target thread splitting the original log file, it first checks whether a file with the first filename exists. If it does, the file with the first filename is deleted, and then the original filename of the original log file is changed to the first filename. If not, the original filename of the original log file is directly changed to the first filename. The first filename can be obtained by adding a first identifier to the original filename. For example, assuming the original filename of the original log file is main.log, the first filename could be main.log1.

[0050] Step 602: Create a file and set the file name of the file to the original file name to obtain the target split file; different target threads obtain different file names for the target split files.

[0051] In this embodiment of the invention, a new file is created and its filename is set to the original filename. This new file with the original filename is then identified as the target split file. Since the filenames of the original log files pointed to by different log paths are different, and the filename of the target split file is the same as the original log filename before the log split, the filenames of the target split files obtained by different target threads will also be different.

[0052] For example, suppose there are two target threads, Thread1 and Thread2. Thread1 processes the original log file "log_20260101.txt". When performing log splitting, it can first rename the original log file to "log_20260101_1.txt", and then create a new file named "log_20260101.txt". After the above operation, the target split file "log_20260101.txt" is obtained. Thread2 processes the original log file "log_20260102.txt". When performing log splitting, it can first rename the original log file to "log_20260102_1.txt", and then create a new file named "log_20260101.txt". After the above operation, the target split file "log_20260102.txt" is obtained.

[0053] In this embodiment of the invention, since each thread has its own independent target split file, the filenames of the target split files corresponding to different threads are different during the log splitting process. This allows for safe and independent subsequent writing and processing operations, avoiding log splitting errors and improving the stability and reliability of the system.

[0054] Optionally, embodiments of the present invention may further include the following steps: Step 701: Encapsulate the specified log of the specified type and the service information corresponding to the specified log to obtain the log to be distributed.

[0055] In this embodiment of the invention, logs of a specified type are selected from the logs received by the second log processor and designated as specified logs. The specified logs of the specified type are then encapsulated with corresponding service information to integrate them according to a predetermined format and structure, resulting in logs to be distributed. The specified type can be set according to actual business needs, and may include different categories such as system operation logs, business operation logs, and error logs. Service information may refer to key information representing the source of the log, such as task identifier, the service name that generated the log, log level, and log type.

[0056] Step 702: Synchronize the logs to be distributed to the distributed message queue; the logs to be distributed are sent to the target cluster via the distributed message queue.

[0057] In this embodiment of the invention, a distributed message queue is pre-created and passed to a second log processor to enable the external delivery of logs. The distributed message queue can be, for example, a Kafka queue. Logs to be distributed are synchronized to the distributed message queue, and the logs stored in the distributed message queue can be sent to the target cluster via the second log processor. The target cluster can be a storage cluster for storing log data, such as a distributed file system or database cluster, or an analysis cluster for log monitoring and analysis. For example, the Kafka module can receive the logs to be distributed sent by the second log processor and then deliver them to the target cluster.

[0058] In this embodiment of the invention, log data decoupling is achieved by synchronizing logs to be distributed to a distributed message queue in the second log processor. That is, the log producer does not need to interact directly with the log consumer. Instead, the second log processor first delivers the logs to be distributed to the message queue, and the message queue is responsible for the subsequent forwarding work. This achieves centralized management of log data and facilitates subsequent operations such as querying, monitoring, statistics, and analysis of logs.

[0059] For example, such as Figure 2As shown, first, a directory (e.g., a log folder) is created to store log files. Two log queues are also created: an error log queue and an information log queue, used to store information logs and error logs respectively. For each log queue, a corresponding first log handler (New Timed RotatingFile Handler) is configured, and these two first log handlers are added to the information log handler and the error log handler, respectively. A second log handler (Multi Queue Handler), such as a multi-queue log handler, is built to manage multiple log queues. A distributed message queue, such as a Kafka queue, is created to send log data to the Kafka cluster. The error log queue, the information log queue, and the distributed message queue are then added to the second log handler.

[0060] Alternatively, in another implementation, multi-threading conflicts can also be avoided through the following steps: Step 801: Before any of the target threads in at least two threads performs a log write operation, obtain the lock state of the locked file based on the target thread; the lock state is used to characterize the current access permissions of the target log file.

[0061] In this embodiment of the invention, the log splitting time can be determined by pre-creating a lock file in the first log processor, and the log write operation can be locked using the lock file to prevent conflicts in multi-threaded log writes. For example, as shown... Figure 3 As shown, the initialization process of the first log processor can be as follows: Create a lock file based on a pre-defined filename. During creation, a read operation can be attempted to open the lock file corresponding to the filename. If the lock file does not exist, it will be automatically created. Check the locking status of the lock file. Since the lock file is initially unlocked, it can be opened. If no other thread is accessing the lock file, the `_do_lock` function can be used to open the lock file, set the corresponding group and permissions, and lock the stream of the lock file to ensure exclusive access to the lock file during initialization. After locking, the log split time point can be read, the next log split time point can be calculated based on the preset log split interval, and the next log split time point can be written to the lock file. After writing the time point, the stream of the lock file can be unlocked, releasing exclusive access to the lock file. Specifically, the locking process can be as follows: Figure 4As shown, when the target thread wants to access the locked file, it checks whether a `stream_lock` (workflow lock) exists in the locked file and whether the workflow lock is closed. If a workflow lock exists and is not locked, the target thread will lock the workflow lock to prevent other threads from using it. If a workflow lock exists but is already locked, it will wait until the workflow lock is unlocked before opening the locked file. If no workflow lock exists, it will open the locked file, create the workflow lock, and then lock it. After the locking operation is completed, the process of reading the log split time point and writing to the next log split time point can be as follows: Figure 5 As shown, the log splitting time point is read from the lock file. Since this is the initialization phase, the lock file does not record any time points; the read log splitting time point is initially 0. When the read time point is 0, the time point for the next log split is calculated based on the preset log splitting interval, and the calculated time point is written to the lock file, providing accurate timing information for subsequent log splitting operations.

[0062] Before any of the target threads performs a log write operation, the lock state of the locked file is acquired based on the target thread. The lock state represents the current access permissions to the target log file. By acquiring the lock state, it can be determined whether other threads are operating on the target log file. The locked file can be a lockfile, used to store the log splitting time. The log splitting time can be a pre-set point in time used to determine whether log splitting is necessary. When the current time exceeds the log splitting time, the log splitting operation is triggered.

[0063] Step 802: When the lock state is unlocked, the lock state of the locked file is switched to locked state by the target thread.

[0064] In this embodiment of the invention, if the lock is in an unlocked state, indicating that no other thread is currently accessing the target log file, the target thread can switch the lock state of the locked file to a locked state, thereby obtaining exclusive access to the target log file and preventing other threads from modifying the file while the thread is writing logs. By locking and unlocking the locked file, access permissions to the target log file can be controlled, preventing multiple threads from operating on the log file simultaneously and causing data corruption.

[0065] Step 803: Obtain the log splitting time stored in the locked file.

[0066] Step 804: If the log splitting time does not meet the log splitting conditions, the log to be written is written to the target log file based on the target thread, and the lock state of the locked file is switched back to the unlock state.

[0067] In this embodiment of the invention, the log splitting time stored in the locked file is obtained. The read log splitting time is compared with the current time to determine whether the log splitting condition is met. If the log splitting time does not exceed the current time, i.e., the log splitting condition is not met, the target thread writes the log to be written to the target log file. After the log writing operation is completed, the lock state of the locked file is switched back to the unlock state, releasing access permissions to the target log file and allowing other threads to perform subsequent operations.

[0068] In this embodiment of the invention, by introducing a file locking and lock state mechanism, synchronization and mutual exclusion between threads are achieved, ensuring that only one thread can operate on the target log file during the log writing and splitting process, thus avoiding data conflicts and loss.

[0069] Optionally, embodiments of the present invention may further include the following steps: Step 901: If the log splitting time meets the log splitting conditions, shut down the workflow of the target log file.

[0070] In this embodiment of the invention, if the current time has elapsed, the log splitting condition is met. Since the locked file has already been switched to a locked state, the workflow for the target log file can be closed, pausing the processing flow related to the current log file. This ensures that other threads cannot operate on the target log file during the log splitting process. For example, closing the workflow can use operations like `_close()` and `__internal_close()` to prevent other threads from deleting the original log file, ensuring data integrity.

[0071] Step 902: The target thread splits the target log file to obtain the target split file.

[0072] In this embodiment of the invention, the target thread performs a splitting operation on the target log file to obtain a target split file. The splitting operation can be performed according to preset rules, such as file size, time interval, etc., to divide the target log file into multiple smaller files for easier subsequent management and storage.

[0073] Step 903: Write the log to be written to the target split file, and switch the locked state of the locked file back to the unlocked state.

[0074] In this embodiment of the invention, the log to be written is written to the target split file. After the log writing operation is completed, the lock state of the locked file is switched back to the unlock state, releasing access permissions to the file, restoring the system to normal, and allowing other threads to operate on the target log file, such as continuing log writing or other related operations. Furthermore, the next log splitting time can be recalculated according to a preset log splitting interval, and the log splitting time in the locked file can be updated.

[0075] In this embodiment of the invention, the need for log splitting is determined based on the log splitting time. When the conditions are met, the target log file is split in a timely manner, which can ensure the reasonable size of the log file and the convenience of management.

[0076] By configuring a locking file to lock each thread during log writing and splitting, the target thread is restricted to performing only log writing or splitting operations when the file is locked. This can negatively impact log writing efficiency under high concurrency. However, in this embodiment of the invention, by binding each thread to a different log file path and log queue, each thread retrieves logs from and writes them only to its assigned log queue. Through independent operation "channels" for threads, queues, and files, conflicts can be avoided while ensuring efficient log writing through parallel multi-threaded writing.

[0077] For example, Figure 6 A schematic diagram of the processing flow of a log processor is shown, such as... Figure 6 As shown, the first log handler, `New Timed Rotating File Handler`, can be configured to inherit the original functionality of `Timed Rotating File Handler` in the `logging.handlers` module, such as time-interval-based log file rotation. The first log handler starts a thread to perform log operations. When the log queue corresponding to the first log handler receives logs distributed by the second log handler, the first log handler retrieves the logs to be written from the queue, filters them, and then performs the write operation. The second log handler, `Multi QueueHandler`, can also be configured to inherit the original functionality of the `logging.handlers` module. After a log write event is triggered, the logs are formatted and then distributed to different log queues (e.g., error log queues and message log queues) based on their log type. Simultaneously, logs of a specified type can be combined with their corresponding service information to obtain logs to be distributed, which are then placed into a distributed message queue (such as a Kafka queue) by the second log handler.

[0078] Figure 7 This is a structural diagram of a log processing device provided in an embodiment of this application, with reference to... Figure 7 The device may include: The first segmentation module 1001 is used to segment the original log file pointed to by the log file path before the target thread performs the log writing operation. If the segmentation information of the log file path bound to the target thread meets the log segmentation conditions, the target thread will segment the original log file to obtain the target segmented file. The target thread is any one of at least two threads, and different threads are bound to different log file paths and log queues. The first acquisition module 1002 is used to acquire logs to be written in the log queue bound to the target thread, based on the target thread. The first writing module 1003 is used to perform a log writing operation on the log to be written to the target split file.

[0079] This invention provides a log processing apparatus. Before a target thread performs a log writing operation, if the partitioning information of the log file path bound to the target thread meets the log partitioning conditions, the target thread partitions the original log file pointed to by the log file path to obtain a target partitioned file. The target thread can be any of at least two threads, each bound to a different log file path and log queue. Based on the target thread, logs to be written in the log queue bound to the target thread are retrieved. Log writing operations are then performed on the logs to be written to the target partitioned file. Because different threads are bound to different log file paths, the original log files of different threads are different. Consequently, the target partitioned files obtained by each thread partitioning the original log file pointed to by its respective log file path are different. Therefore, in multi-threaded log writing and partitioning scenarios, log writing conflicts can be avoided, ensuring the integrity of log data and the reliability of log processing. Simultaneously, because each thread is bound to a different log queue, each thread can execute in parallel, writing the logs to be written in the log queue to its respective target partitioned file. This avoids log writing conflicts while ensuring the parallelism of the solution, thereby ensuring log writing efficiency.

[0080] Optionally, the device further includes: The second acquisition module is used to acquire logs to be written from the log queue bound to the target thread when the partitioning information of the log file path corresponding to the target thread does not meet the log partitioning conditions. The second writing module is used to perform a log writing operation on the log to be written to the original log file.

[0081] Optionally, the device further includes: The first creation module is used to create a log queue corresponding to any log type. The second creation module is used to create a thread corresponding to any of the log queues and an original log file corresponding to the thread. The first binding module is used to bind the log file path corresponding to the original log file to the thread; different log file paths point to different file names of the original log files.

[0082] Optionally, the device further includes: The first determining module is used to determine, in the event of a triggered log write event, a log queue that matches the log type indicated by the log write event as the log queue to be distributed, based on the mapping relationship between log type and log queue. The first distribution module is used to distribute the log indicated by the log write event to the log queue to be distributed.

[0083] Optionally, the second creation module includes: The first creation submodule is used to create a first log processor corresponding to any of the log queues. The second creation submodule is used to create a thread corresponding to the log queue through the first log processor; The first distribution module includes: The first distribution submodule is used to distribute the logs indicated by the log write event to the log queue to be distributed via the second log processor.

[0084] Optionally, the first segmentation module 1001 includes: The first modification module is used by the target thread to change the original file name of the original log file to a first file name; the first file name is obtained by adding a first identifier to the original file name. The first setting module is used to create a file and set the file name of the file to the original file name to obtain the target split file; different target threads obtain different file names for the target split files.

[0085] Optionally, the device further includes: The first encapsulation module is used to encapsulate a specified log of a specified type and the service information corresponding to the specified log to obtain a log to be distributed. The first synchronization module is used to synchronize the logs to be distributed to a distributed message queue; the logs to be distributed are sent to the target cluster via the distributed message queue.

[0086] Optionally, the device further includes: The third acquisition module is used to acquire the log splitting time stored in the locked file before either of the target threads in at least two threads performs a log write operation. The fourth acquisition module is used to acquire the lock status of the locked file based on the target thread when the log splitting time does not meet the log splitting conditions; the lock status is used to represent the current access permissions of the target log file. The first switching module is used to switch the lock state of the locked file to the locked state through the target thread when the lock state is unlocked. The second switching module is used to write the log to be written to the target log file based on the target thread, and switch the lock state of the locked file back to the unlock state.

[0087] Optionally, the device further includes: The third switching module is used to switch the lock state of the locked file to the locked state and shut down the workflow of the target log file when the log splitting time meets the log splitting conditions. The second segmentation module is used to segment the target log file by the target thread to obtain the target segmented file; The fourth switching module is used to write the log to be written to the target split file and switch the locked state of the locked file back to the unlocked state.

[0088] This invention also provides an electronic device, such as... Figure 8 As shown, it includes a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104. The processor 1101, communication interface 1102, and memory 1103 communicate with each other via the communication bus 1104. Memory 1103 is used to store computer programs; When processor 1101 executes the program stored in memory 1103, it performs the following steps: Before the target thread performs the log writing operation, if the splitting information of the log file path bound to the target thread meets the log splitting conditions, then the target thread splits the original log file pointed to by the log file path to obtain the target split file; the target thread is any one of at least two threads, and different threads are bound to different log file paths and log queues; Based on the target thread, retrieve the logs to be written from the log queue bound to the target thread; Perform a log writing operation on the log to be written to the target split file.

[0089] Optionally, the method further includes: If the log file path segmentation information corresponding to the target thread does not meet the log segmentation conditions, logs to be written are obtained from the log queue bound to the target thread based on the target thread. Perform a log write operation on the log to be written to the original log file.

[0090] Optionally, the method further includes: For any log type, create a log queue corresponding to that log type; For any of the log queues, create the thread corresponding to the log queue and the original log file corresponding to the thread; The log file path corresponding to the original log file is bound to the thread; different log file paths point to different filenames of the original log files.

[0091] Optionally, the method further includes: When a log write event is triggered, based on the mapping relationship between log type and log queue, the log queue that matches the log type indicated by the log write event is determined as the log queue to be distributed. The log written by the event is distributed to the log queue to be distributed.

[0092] Optionally, creating a thread corresponding to any of the log queues includes: For any of the log queues, create a first log processor corresponding to the log queue; The thread corresponding to the log queue is created through the first log processor; The step of distributing the log indicated by the log write event to the log queue to be distributed includes: The log written by the second log processor is distributed to the log queue to be distributed.

[0093] Optionally, the step of splitting the original log file pointed to by the log file path by the target thread to obtain the target split file includes: The target thread changes the original filename of the original log file to a first filename; the first filename is obtained by adding a first identifier to the original filename. Create a file and set its filename to the original filename to obtain the target split file; different target threads obtain different filenames for the target split files.

[0094] Optionally, the method further includes: Encapsulate the specified log of the specified type and the corresponding service information of the specified log to obtain the log to be distributed; The logs to be distributed are synchronized to a distributed message queue; the logs to be distributed are then sent to the target cluster via the distributed message queue.

[0095] Optionally, the method further includes: Before any of the target threads in at least two threads performs a log write operation, the lock state of the locked file is obtained based on the target thread; the lock state is used to characterize the current access permissions of the target log file. When the lock is in the unlocked state, the target thread switches the lock state of the locked file to the locked state. Retrieve the log splitting time stored in the locked file; If the log splitting time does not meet the log splitting conditions, the log to be written is written to the target log file based on the target thread, and the lock state of the locked file is switched back to the unlock state.

[0096] Optionally, the method further includes: If the log splitting time meets the log splitting conditions, the workflow of the target log file is shut down; The target thread splits the target log file to obtain the target split file; Write the log to be written to the target split file, and switch the locked state of the locked file back to the unlocked state.

[0097] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0098] The communication interface is used for communication between the aforementioned terminal and other devices.

[0099] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0100] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0101] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the log processing methods described in the above embodiments.

[0102] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the log processing methods described in the above embodiments.

[0103] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0104] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0105] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0106] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A log processing method, characterized in that, The method includes: Before the target thread performs the log writing operation, if the splitting information of the log file path bound to the target thread meets the log splitting conditions, then the target thread splits the original log file pointed to by the log file path to obtain the target split file; the target thread is any one of at least two threads, and different threads are bound to different log file paths and log queues; Based on the target thread, retrieve the logs to be written from the log queue bound to the target thread; Perform a log writing operation on the log to be written to the target split file.

2. The method according to claim 1, characterized in that, The method further includes: If the log file path segmentation information corresponding to the target thread does not meet the log segmentation conditions, logs to be written are obtained from the log queue bound to the target thread based on the target thread. Perform a log write operation on the log to be written to the original log file.

3. The method according to claim 1, characterized in that, The method further includes: For any log type, create a log queue corresponding to that log type; For any of the log queues, create the thread corresponding to the log queue and the original log file corresponding to the thread; The log file path corresponding to the original log file is bound to the thread; different log file paths point to different filenames of the original log files.

4. The method according to claim 3, characterized in that, The method further includes: When a log write event is triggered, based on the mapping relationship between log type and log queue, the log queue that matches the log type indicated by the log write event is determined as the log queue to be distributed. The log written by the event is distributed to the log queue to be distributed.

5. The method according to claim 4, characterized in that, The step of creating a thread corresponding to any of the log queues includes: For any of the log queues, create a first log processor corresponding to the log queue; The thread corresponding to the log queue is created through the first log processor; The step of distributing the log indicated by the log write event to the log queue to be distributed includes: The log written by the second log processor is distributed to the log queue to be distributed.

6. The method according to claim 1, characterized in that, The process of splitting the original log file pointed to by the log file path by the target thread to obtain the target split file includes: The target thread changes the original filename of the original log file to a first filename; the first filename is obtained by adding a first identifier to the original filename. Create a file and set its filename to the original filename to obtain the target split file; different target threads obtain different filenames for the target split files.

7. The method according to claim 1, characterized in that, The method further includes: Encapsulate the specified log of the specified type and the corresponding service information of the specified log to obtain the log to be distributed; The logs to be distributed are synchronized to a distributed message queue; the logs to be distributed are then sent to the target cluster via the distributed message queue.

8. A log processing device, characterized in that, The device includes: The first segmentation module is used to segment the original log file pointed to by the log file path before the target thread performs the log writing operation. If the segmentation information of the log file path bound to the target thread meets the log segmentation conditions, the target thread will segment the original log file to obtain the target segmented file. The target thread is any one of at least two threads, and different threads are bound to different log file paths and log queues. The first acquisition module is used to acquire logs to be written from the log queue bound to the target thread, based on the target thread. The first writing module is used to perform a log writing operation on the log to be written to the target split file.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images