Method and apparatus for intercepting and processing multi-threaded log, and electronic device

Abstract

The application relates to a multi-thread log interception and processing method and device and electronic equipment. The method comprises the following steps: in an algorithm task execution process, unified interception is performed on log information generated by standard output, uncaptured exceptions, log modules and third-party log libraries, and corresponding standardized log record objects are generated through a log distribution recorder; a log filter is called, and the log record objects are filtered and processed according to preset filtering rules; according to thread identification information carried in the log record objects, the thread distribution processor is used to distribute the log record objects to corresponding file processors; the file processor is used to call a log formatter, the log record objects are formatted, log output content conforming to a unified format template is generated, and the log output content is written into corresponding log files. The application solves the technical problems of log format fragmentation, lack of task association and high integration cost.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, and electronic device for intercepting and processing multi-threaded logs. Background Technology

[0002] With the rapid development of artificial intelligence technology, algorithm service-oriented architecture has become an important component of modern software systems and is widely used in scenarios such as large-scale algorithm service platforms, multi-threaded concurrent processing, production environment troubleshooting, and log auditing and analysis. In algorithm service-oriented production environments, algorithm tasks typically involve multiple stages, including data preprocessing, model inference, and result post-processing. Each stage may generate various types of log outputs, and the diversity of log sources and output methods leads to fragmented logs that are difficult to manage and analyze uniformly. Existing technologies struggle to achieve unified interception, formatting, and standardized output of non-standard log sources (such as print statements or system exception information) without modifying business code, thus increasing the difficulty of log parsing and analysis. Furthermore, existing log systems lack task-level association mechanisms, failing to automatically associate logs in multi-task concurrent environments with corresponding task identifiers (jobId). Under massive concurrent tasks, log location and query efficiency is low and error-prone. In addition, existing log systems have high integration costs, typically requiring explicit input of context information at each log output point, resulting in complex maintenance and a high risk of omissions, and failing to achieve non-intrusive log interception and automatic injection. The aforementioned technical issues severely restrict the uniformity, maintainability, and troubleshooting efficiency of log management in an algorithm-service environment. Summary of the Invention

[0003] This application provides a method, apparatus, and electronic device for intercepting and processing multi-threaded logs to solve the technical problems of fragmented log formats, lack of task association, and high integration costs.

[0004] Firstly, this application provides a method for intercepting and processing multi-threaded logs, comprising: during the execution of an algorithm task, uniformly intercepting log information generated by standard output, uncaught exceptions, the logging module, and third-party logging libraries, and sending the intercepted log information to a log dispatcher, thereby generating a corresponding standardized log record object; passing the log record object to its associated log processor through the log dispatcher, and calling a log filter to filter the log record object according to preset filtering rules; if the log record object passes the preset filtering rules, distributing the log record object to its corresponding file processor according to the thread identifier information carried in the log record object through the thread dispatcher; and calling a log formatter through the file processor to format the log record object, generating log output content conforming to a unified format template, and writing the log output content to its corresponding log file.

[0005] Secondly, this application provides a multi-threaded log interception and processing device, comprising: an interception module, used to uniformly intercept log information generated by standard output, uncaught exceptions, log modules, and third-party log libraries during the execution of an algorithm task, and send the intercepted log information to a log distribution recorder, through which a corresponding standardized log record object is generated; a filtering module, used to pass the log record object to its associated log processor through the log distribution recorder, and call a log filter to filter the log record object according to a preset filtering rule; a distribution module, used to distribute the log record object to its corresponding file processor through the thread distribution processor according to the thread identifier information carried in the log record object if the log record object passes the preset filtering rule; and an output module, used to call a log formatter through the file processor to format the log record object, generate log output content conforming to a unified format template, and write the log output content to its corresponding log file.

[0006] As an optional example, the above device further includes: a first initialization module, used to perform a global initialization operation when the above algorithm task is executed for the first time and log interception is started before uniformly intercepting the log information generated by the standard output, uncaught exceptions, logging module and third-party logging library; wherein the above global initialization operation is executed only once in the entire process lifecycle; and a second initialization module, used to perform a log instance initialization operation corresponding to the thread when a new thread is detected to generate log output for the first time after the above global initialization operation is completed.

[0007] As an optional example, the first initialization module includes: a first acquisition unit, configured to acquire or create a log dispatch logger and configure a log level for the log dispatch logger; a first creation unit, configured to create a thread dispatch processor associated with the log dispatch logger and register the thread dispatch processor as a log processor of the log dispatch logger; and a second creation unit, configured to create a log filter and a log formatter and configure the log filter and the log formatter to the thread dispatch processor to form a complete log processing pipeline.

[0008] As an optional example, the second initialization module includes: a storage unit for storing the log file path or log identification information corresponding to the algorithm task into a thread context variable; a checking unit for checking whether the global initialization operation has been completed, and triggering the execution flow of the global initialization operation if the global initialization operation has not been completed; and an execution unit for performing only the initialization and storage operations of thread-local data if the global initialization operation has been completed.

[0009] As an optional example, the interception module includes: a parsing unit, used to perform preliminary parsing on the intercepted log information to extract the source of the log information, wherein the source of the log information includes the log level, timestamp, thread identifier, source file path, line number, and log message content; and a standardization unit, used to map or standardize the log level according to the source of the log information to generate a corresponding standardized log record object.

[0010] As an optional example, the filtering module includes: a second acquisition unit, configured to acquire the preset filtering rules, wherein the preset filtering rules include an exclusion path list, an exclusion message list, a inclusion path list, and a inclusion message list; a conversion unit, configured to extract the message content, source file path, and related context information from the log recording object, and convert the message content and source file path of the log recording object into a corresponding comparison form according to a preset case sensitivity configuration; and a removal unit, configured to determine that the log recording object has not passed the preset filtering rules and remove the log recording object from the log processing pipeline if the source file path of the log recording object begins with any exclusion path in the exclusion path list, or the message content of the log recording object contains any exclusion keyword in the exclusion message list, or the source file path of the log recording object does not begin with any inclusion path in the inclusion path list, or the message content of the log recording object does not contain any inclusion keyword in the inclusion message list.

[0011] As an optional example, the above output module includes: an extraction unit for extracting log information sources from the above log record object; and an arrangement unit for arranging the above log information sources according to the above uniform format template, and merging the fixed text defined in the above uniform format template with the variable information in the above log record object to generate corresponding log output content.

[0012] Thirdly, this application provides a storage medium storing a computer program, wherein the computer program is executed by a processor to perform the above-described method for intercepting and processing multi-threaded logs.

[0013] Fourthly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described method for intercepting and processing multi-threaded logs through the computer program.

[0014] The technical solutions provided in this application have the following advantages compared with the prior art: This application employs a method to uniformly intercept log information generated by standard output, uncaught exceptions, the logging module, and third-party logging libraries during algorithm task execution. The intercepted log information is then sent to a log dispatcher, which generates corresponding standardized log record objects. These log record objects are then passed to their associated log processors, and log filters are invoked to filter them according to preset filtering rules. If a log record object passes the preset filtering rules, a thread dispatcher distributes it to its corresponding file processor based on the thread identifier information carried within the log record object. Finally, the file processor invokes a log formatter to format the log record object, generating a format that conforms to the specified format. The method of unifying the log output content of a template and writing the log output content to the corresponding log file achieves unified management of logs in a multi-task concurrent environment, automatic task-level association, and physical isolation storage. This improves log readability, analyzability, and troubleshooting efficiency in the production environment, thereby solving the technical problems of fragmented log formats, lack of task association, and high integration costs. This is achieved by uniformly intercepting standard output, uncaught exceptions, log information from the log module and third-party log libraries generated during algorithm task execution, sending them to a log dispatcher to generate standardized log record objects, filtering them through preset rules of the log filter, and then distributing them to the corresponding file processors based on thread identifiers. The log formatter then generates a unified format output and writes it to the corresponding log file. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0018] Figure 1 This is a flowchart of an optional method for intercepting and processing multi-threaded logs according to an embodiment of this application; Figure 2 This is a flowchart illustrating the specific implementation of an optional multi-threaded log interception and processing method according to an embodiment of this application. Figure 3 This is a schematic diagram of the structure of an optional multi-threaded log interception and processing device according to an embodiment of this application; Figure 4 This is a schematic diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0021] According to a first aspect of the embodiments of this application, a method for intercepting and processing multi-threaded logs is provided, optionally, as follows: Figure 1 As shown, the above method includes: S102, during the execution of the algorithm task, uniformly intercepts the log information generated by the standard output, uncaught exceptions, log module and third-party log library, and sends the intercepted log information to the log distribution recorder, which generates the corresponding standardized log record object. S104, the log recording object is passed to the log processor associated with it through the log dispatcher, and the log filter is invoked to filter the log recording object according to the preset filtering rules; S106, If the log recording object passes the preset filtering rules, the thread dispatcher dispatches the log recording object to the corresponding file processor according to the thread identifier information carried in the log recording object; S108 calls the log formatter through the file processor to format the log record object, generate log output content that conforms to a unified format template, and write the log output content to its corresponding log file.

[0022] Optionally, this embodiment provides a method for intercepting and processing multi-threaded logs, aiming to solve problems such as fragmented log formats, lack of task-level correlation, and high log integration costs in algorithm service scenarios. It is applicable to multi-threaded concurrent scenarios involved in algorithm task execution, and can achieve unified interception and centralized management of log information generated by standard output, uncaught exceptions, log modules, and third-party log libraries, thereby ensuring that logs from different sources and of different types can be output in a standardized, traceable, and task-level correlated manner within a unified processing system.

[0023] In specific implementations, such as Figure 2 The flowchart shown illustrates the specific implementation process. When an algorithm task is executed, the system uses a zero-intrusion interception mechanism to uniformly access various log information generated during the task's execution through the log distribution recorder. The log distribution recorder, acting as the unified entry point for the log processing pipeline, parses the received log information and generates corresponding standardized log record objects. Standardization processing includes extracting key information such as log level, thread identifier, source file path, line number, timestamp, and log message content. Simultaneously, it maps or unifies log levels from different log sources, ensuring consistent level semantics across multi-threaded, multi-module, and multi-source log output. The generated standardized log record objects serve as the core carrier of the log processing pipeline, triggering the execution of subsequent processor chains.

[0024] Subsequently, the log object is passed to its associated log handler, and the log filter is invoked to execute preset filtering rules. The filter evaluates the logs based on the message content, source file path, and other contextual information within the log object, combined with inclusion and exclusion rules. Specifically, this includes: iterating through the exclusion path and exclusion keyword list to remove unnecessary logs; iterating through the inclusion path and inclusion keyword list to filter out logs that need to be retained; and supporting case-sensitive configuration to ensure consistency of filtering operations across different environments. Only log objects that pass the filter will proceed to the next step of processing.

[0025] After a log record object passes the preset filtering rules, the thread dispatcher distributes it to the corresponding file processor based on the thread identifier information carried in the log record object. Each thread corresponds to a unique file processor and log file, ensuring physical isolation and security of multi-threaded log output. Upon receiving a log record object, the file processor formats it by calling the log formatter, outputting the log information according to a unified format template, including timestamp display format, log level identifier, thread ID, source file path, and line number, ensuring consistency in format and structure across logs from different sources. The formatted log string is ultimately written to the log file corresponding to that thread, supporting file cache refresh, periodic writing, or write policy control to ensure the integrity and real-time performance of the log data.

[0026] The above method achieves unified interception, filtering, formatting, and distribution of multi-threaded logs for algorithm tasks, enabling logs from different sources to be automatically associated with corresponding task identifiers. This solves the problems of fragmented log formats, lack of task-level association, and high integration costs in existing technologies. It offers advantages such as zero intrusion, scalability, thread safety, accurate task tracking, and high log readability and analyzability, significantly improving the log management efficiency of large-scale algorithm service platforms and the troubleshooting capabilities of production environments.

[0027] As an optional example, before uniformly intercepting log information generated by standard output, uncaught exceptions, logging modules, and third-party logging libraries, the above method also includes: When the algorithm task is executed for the first time and log interception is started, a global initialization operation is performed. This global initialization operation is executed only once during the entire process lifecycle. After completing the global initialization operation, when a new thread is detected to generate log output for the first time, the log instance initialization operation corresponding to that thread is executed.

[0028] Optionally, in this embodiment, a global initialization operation is performed when the algorithm task is executed for the first time and log interception is initiated. The global initialization operation includes creating a log dispatcher, thread dispatcher, log filter, and log formatter, and configuring necessary parameters such as log level settings, format template settings, and filter rule loading to ensure that the entire log processing pipeline can work uniformly in subsequent algorithm tasks. To conserve system resources and ensure the consistency of global components, this global initialization operation is executed only once during the entire process lifecycle, and a singleton pattern and initialization lock mechanism are used to prevent duplicate creation or conflicts in a multi-threaded environment.

[0029] After completing the global initialization, when the system detects a new thread generating log output for the first time, it performs log instance initialization for that thread. Thread-level instance initialization includes allocating an independent file handler for the thread, generating a corresponding log file path, and storing relevant thread-local information in the thread context variable (ContextVar). This allows subsequent log output processes to automatically identify the thread identifier, achieving thread-isolated output. Thread instance initialization is only performed on new threads and does not trigger global initialization repeatedly, thus ensuring the consistency of global resource reuse and thread-level independent management. In this way, each thread corresponding to an algorithm task can share the globally unified log processing component while maintaining the physical isolation of its own log files, achieving security, traceability, and efficient management of multi-threaded log output.

[0030] By combining global initialization and thread instance initialization mechanisms, the log processing pipeline achieves high availability and scalability. This not only ensures unified log management and task-level correlation in a multi-threaded environment, but also reduces system resource consumption and the risk of repeated initialization, thereby improving the efficiency and reliability of log collection, analysis, and troubleshooting in large-scale algorithm service environments.

[0031] As an optional example, performing global initialization operations includes: Obtain or create a log distribution logger, and configure the log level for the log distribution logger; Create a thread dispatcher associated with the log dispatcher and register the thread dispatcher as a log handler for the log dispatcher; Create log filters and log formatters, and configure them to thread dispatch processors to form a complete log processing pipeline.

[0032] Optionally, in this embodiment, a log distribution logger is acquired or created. This logger serves as the unified entry point for the entire log processing pipeline, receiving all log information from standard output, uncaught exceptions, the standard logging module, and third-party logging libraries. After creating the log distribution logger, the system configures its log levels to ensure that logs from different sources are processed and classified according to a unified level standard in the pipeline, thereby achieving standardized log management.

[0033] Subsequently, the system creates a thread dispatcher associated with the log dispatcher and registers this thread dispatcher as the log handler for the log dispatcher. The main function of the thread dispatcher is to dispatch logs to the corresponding thread's file handler based on the thread identifier information carried in the log object, achieving physical isolation and thread-level independent management of log output in a multi-threaded environment. In this step, the thread dispatcher maintains the correspondence between thread IDs and file handlers through an internal mapping table, ensuring that each thread's logs can be correctly and independently written to its corresponding log file.

[0034] After creating and registering the thread dispatcher, log filters and log formatters are further created. Log filters are used to select log entries based on preset inclusion and exclusion rules, ensuring that only logs meeting the criteria enter the output process. Log formatters are used to generate standardized log strings from key information in the log entry (including timestamp, log level, thread ID, source file path, and message content) according to a unified format template. Subsequently, the log filters and log formatters are configured to the thread dispatcher, forming a complete log processing pipeline, enabling unified management and control of all stages from log interception to output.

[0035] As an optional example, performing the log instance initialization operation corresponding to this thread includes: Store the log file path or log identifier information corresponding to the algorithm task into the thread context variable; Check whether the global initialization operation has been completed, and if it is detected that the global initialization operation has not been completed, trigger the execution flow of the global initialization operation; If global initialization has been detected as complete, only the initialization and storage of thread-local data will be performed.

[0036] Optionally, in this embodiment, the log file path, task identification information (such as jobId), or thread unique identification information corresponding to the algorithm task are first stored in a thread context variable (ContextVar). This allows the system to automatically obtain thread-local information during subsequent log interception, distribution, and output, achieving task-level association and thread-isolated storage of logs. Through the thread context variable mechanism, the system can ensure that the log data of each thread is independent and does not interfere with other threads in a multi-threaded environment, while also ensuring the automatic binding of log records to specific task identifiers.

[0037] Subsequently, the system checks whether the global initialization operation has been completed. If global initialization is not yet complete, thread instance initialization triggers the execution flow of the global initialization operation. Global initialization includes creating log dispatch loggers, thread dispatch processors, log filters, and log formatters, and configuring and registering necessary parameters to ensure the uniformity and reusability of the entire log processing pipeline. Through this mechanism, thread instance initialization can dynamically trigger the creation of global components while ensuring thread independence, ensuring that the log processing pipeline runs uniquely and stably throughout the entire process lifecycle.

[0038] If global initialization is detected as complete, thread instance initialization only performs thread-local data initialization and storage operations, without recreating global components. This includes assigning independent file handlers to threads, generating corresponding log file paths, and storing this information in the thread context so that the log dispatcher can correctly distribute log recording objects based on the thread ID. Through this design, each thread can share a globally unified log processing component while maintaining physical isolation and independent management of log files, thereby achieving security, traceability, and efficient management of log output in a multi-task concurrent environment.

[0039] As an optional example, the intercepted log information is sent to the log dispatcher, which generates corresponding standardized log recording objects, including: The intercepted log information is initially parsed to extract the source of the log information, which includes the log level, timestamp, thread identifier, source file path, line number, and log message content. Based on the source of the log information, the log level is mapped or standardized to generate the corresponding standardized log record object.

[0040] Optionally, in this embodiment, the intercepted log information is first preliminarily parsed to extract its source. The source of the log information includes, but is not limited to, key information such as log level, timestamp, thread identifier, source file path, line number, and log message content. By parsing this information, the system can accurately identify the source of the log, the thread to which it belongs, and the context of the log recording, providing basic data for subsequent thread distribution and task-level association.

[0041] Subsequently, the system maps or standardizes log levels based on the extracted log information source. Since different log sources may use different log level naming rules or granularities—for example, the log levels of standard output, the Python logging module, and third-party logging libraries may be inconsistent—the log dispatcher uses mapping rules to uniformly convert them into the system's internally agreed-upon standard log levels, ensuring log level consistency throughout the entire processing pipeline. This standardization process not only includes log levels but also unifies timestamp formats, ensuring consistent readability and analyzability of logs across threads, tasks, and sources.

[0042] After completing log level mapping and normalization, the log dispatcher generates a normalized log record object. This object contains all the key information of the log, including the normalized log level, timestamp, thread identifier, source file path, line number, and log message content. As the core data structure of the log processing pipeline, the normalized log record object can be passed to log filters for filtering, thread dispatchers for thread-level distribution, and file processors and formatters for final output.

[0043] As an optional example, invoking the log filter to filter log records according to preset filtering rules includes: Obtain preset filtering rules, which include an exclusion path list, an exclusion message list, an inclusion path list, and an inclusion message list. Extract the message content, source file path, and related context information from the log record object, and convert the message content and source file path of the log record object into the corresponding comparison form according to the preset case sensitivity configuration; If the source file path of the log object starts with any of the excluded paths in the excluded path list, or the message content of the log object contains any of the excluded keywords in the excluded message list, or the source file path of the log object does not start with any of the included paths in the included path list, or the message content of the log object does not contain any of the included keywords in the message list, then the log object is determined to have failed the preset filtering rules and is removed from the log processing pipeline.

[0044] Optionally, in this embodiment, the system first obtains preset filtering rules, which include information such as an exclusion path list, an exclusion message list, an inclusion path list, and an inclusion message list, used to perform conditional filtering on log recording objects. The preset filtering rules can also be configured with case sensitivity, so that when comparing source file paths and message content, a case-sensitive or case-insensitive comparison method can be selected according to specific needs, thereby achieving a flexible log filtering strategy.

[0045] Subsequently, the system extracts key information from the log record object, including the log message content, source file path, and related context information. Based on the preset case sensitivity configuration, the message content and source file path of the log record object are converted into corresponding comparison forms to match the preset inclusion and exclusion rules. Through this conversion, the system can maintain the effectiveness and consistency of log filtering rules across different log sources, modules, and encoding formats.

[0046] After extracting and converting log information, the log filter performs rule matching on the log record objects. If the source file path of a log record object begins with any excluded path in the excluded path list, or if the log message content contains any excluded keyword in the excluded message list, the system will determine that the log record object does not meet the preset rules and remove it from the log processing pipeline. Similarly, if the source file path of a log record object does not begin with any included path in the included path list, or if the log message content does not contain any included keyword in the message list, the log record object will also be determined to have failed the filter and will be rejected.

[0047] As an optional example, the log formatter is invoked through the file processor to format the log record object and generate log output content that conforms to a uniform format template, including: Extract the source of log information from the log record object; Based on the uniform format template, the source of log information is formatted and arranged, and the fixed text defined in the uniform format template is merged with the variable information in the log record object to generate the corresponding log output content.

[0048] Optionally, in this embodiment, the file processor first extracts the source of log information from the log recording object, including key information such as log level, timestamp, thread identifier, source file path, line number, and log message content. This information reflects the source, generation environment, and context of the log, and is the core data for log output and analysis.

[0049] Subsequently, the system formats the extracted log information sources according to a uniform formatting template. The uniform formatting template defines the standard structure of log output, including a fixed text section and placeholders for displaying variable information. The log formatter merges the variable information from the log record object with the fixed text in the template, generating a standardized log output string by arranging key information such as timestamps, log levels, thread IDs, source file paths, and log messages in template order. This process not only ensures consistency in format for logs from different sources and of different types but also facilitates automatic log parsing, searching, and analysis.

[0050] During the generation of log output, the log formatter can also perform extended processing based on template configuration, such as adding task identifiers (jobId), log category tags, or custom context information to further improve log readability and traceability. Through unified formatting, the system can ensure that the log output structure of each thread is consistent in a multi-threaded, multi-task concurrent environment, while maintaining task-level correlation and thread-level isolation.

[0051] The final generated log output is written to the corresponding log file by the file processor, realizing a complete processing pipeline from log interception, filtering, thread distribution to formatted output. This method not only solves the log format fragmentation problem but also improves the readability, analyzability, and troubleshooting efficiency of logs in large-scale algorithm service scenarios, while ensuring a unified standard for log output and security and stability in multi-threaded environments.

[0052] To illustrate with an example, this application achieves centralized management and multi-threaded isolated output of logs from multiple sources and of multiple types through a unified log interception and processing mechanism. The processing flow is as follows: 1. Standard Output Interception: When the algorithm code executes a `print` statement, the system captures log information by redirecting standard output and calls the `log` method of the log dispatcher, passing in the log level (INFO) and message string. The `stacklevel` parameter ensures that the original file path and line number of the `print` statement are preserved in the log record. The log dispatcher passes the log record to the registered handler (thread dispatcher). The thread dispatcher finds the corresponding file handler based on the thread ID in the log record and calls its `emit` method to output the log. The file handler then calls the log formatter to format the log, generating a log string that conforms to a uniform format and writing it to the corresponding log file.

[0053] 2. Exception Log Capture: When an uncaught exception occurs during algorithm code execution, the system captures the exception by redirecting a Python system-level exception hook function. The exception handling function receives the exception type, exception value, and exception backtracking object. The system extracts the file path, line number, and function name of the last stack frame from the backtracking object and overwrites the relevant information in the log record. Subsequently, the `error` method of the log dispatcher is called to generate a log record object from the exception information and path overwrite information. After being filtered and formatted, the object is dispatched to the file handler of the corresponding thread and output to the log file.

[0054] 3. Standard Log Module Processing: When the algorithm code uses Python's standard logging module to output logs, the logging module internally creates a LogRecord object, which contains a timestamp, thread ID, file path, line number, log level, and message content. The LogRecord object is first filtered by registered filters. Logs that meet the rules are dispatched by the thread dispatcher to the corresponding file handler based on the thread ID. Then, a formatter is called to generate standardized log output, which is written to the corresponding log file for each thread.

[0055] 4. Third-Party Log Library Bridging: For third-party log libraries (taking Loguru as an example), log records are first sent to registered sink functions, including system-defined sink functions. Custom sink functions extract information such as log level, message content, file path, and line number from the log record object and call the log method of the log dispatcher to forward the log to the standard logging system. After filtering, formatting, and thread dispatch processing by the standard logging system, the log is finally output to the corresponding thread's log file, achieving unified management with the standard log.

[0056] Through the above process, the system achieves unified interception, filtering, formatting, and thread-distributed output of print output, uncaught exceptions, standard log modules, and third-party log libraries, ensuring task-level correlation, physical isolation, and format consistency of logs in a multi-threaded environment.

[0057] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0058] According to another aspect of the embodiments of this application, a device for intercepting and processing multi-threaded logs is also provided, such as... Figure 3 As shown, it includes: The interception module 302 is used to uniformly intercept log information generated by standard output, uncaught exceptions, log module and third-party log library during the execution of algorithm task, and send the intercepted log information to the log distribution recorder, which generates corresponding standardized log record objects. The filtering module 304 is used to pass the log recording object to the associated log processor through the log dispatcher and call the log filter to filter the log recording object according to the preset filtering rules. The distribution module 306 is used to distribute the log recording object to its corresponding file processor according to the thread identification information carried in the log recording object when the log recording object passes the preset filtering rules. The output module 308 is used to call the log formatter through the file processor to format the log record object, generate log output content that conforms to a unified format template, and write the log output content to its corresponding log file.

[0059] It should be noted that the interception module 302 in this embodiment can be used to execute step S102 in this application embodiment, the filtering module 304 in this embodiment can be used to execute step S104 in this application embodiment, the distribution module 306 in this embodiment can be used to execute step S106 in this application embodiment, and the output module 308 in this embodiment can be used to execute step S108 in this application embodiment.

[0060] As an optional example, the above-described apparatus further includes: The first initialization module is used to perform a global initialization operation when the algorithm task is executed for the first time and log interception is started, before uniformly intercepting the log information generated by the standard output, uncaught exceptions, logging module and third-party logging library. The global initialization operation is executed only once in the entire process lifecycle. The second initialization module is used to perform the log instance initialization operation corresponding to a new thread when a new thread is detected to generate log output for the first time after the global initialization operation is completed.

[0061] As an optional example, the first initialization module includes: The first acquisition unit is used to acquire or create a log distribution logger and configure the log level for the log distribution logger. The first creation unit is used to create a thread dispatch processor associated with the log dispatch logger and register the thread dispatch processor as a log processor of the log dispatch logger. The second creation unit is used to create log filters and log formatters, and configure the log filters and log formatters to the thread dispatch processor to form a complete log processing pipeline.

[0062] As an optional example, the second initialization module includes: The storage unit is used to store the log file path or log identification information corresponding to the algorithm task into the thread context variable; The checking unit is used to check whether the global initialization operation has been completed, and if it is detected that the global initialization operation has not been completed, it triggers the execution flow of the global initialization operation. The execution unit is used to perform only the initialization and storage operations of thread-local data when it is detected that the global initialization operation has been completed.

[0063] As an optional example, the interception module includes: The parsing unit is used to perform preliminary parsing on the intercepted log information to extract the source of the log information. The source of the log information includes the log level, timestamp, thread identifier, source file path, line number, and log message content. The standardization unit is used to map or standardize log levels based on the source of log information, and generate corresponding standardized log record objects.

[0064] As an optional example, the filtering module includes: The second acquisition unit is used to acquire preset filtering rules, wherein the preset filtering rules include an exclusion path list, an exclusion message list, an inclusion path list, and an inclusion message list. The conversion unit is used to extract the message content, source file path and related context information from the log record object, and convert the message content and source file path of the log record object into the corresponding comparison form according to the preset case sensitivity configuration; The removal unit is used to determine that a log object has not passed the preset filtering rules and remove the log object from the log processing pipeline if the source file path of the log object starts with any excluded path in the excluded path list, or the message content of the log object contains any excluded keyword in the excluded message list, or the source file path of the log object does not start with any included path in the included path list, or the message content of the log object does not contain any included keyword in the message list.

[0065] As an optional example, the output module includes: The extraction unit is used to extract the source of log information from the log record object; The arrangement unit is used to arrange the log information sources in a format according to a uniform format template, and to merge the fixed text defined in the uniform format template with the variable information in the log record object to generate the corresponding log output content.

[0066] For other examples of this embodiment, please refer to the examples above, which will not be repeated here.

[0067] Figure 4 This is a schematic diagram of an optional electronic device according to an embodiment of this application, such as... Figure 4 As shown, it includes a processor 402, a communication interface 404, a memory 406, and a communication bus 408. The processor 402, communication interface 404, and memory 406 communicate with each other via the communication bus 408. Memory 406 is used to store computer programs; When processor 402 executes a computer program stored in memory 406, it performs the following steps: During the execution of the algorithm task, the log information generated by the standard output, uncaught exceptions, logging module and third-party logging library is uniformly intercepted, and the intercepted log information is sent to the log distribution recorder, which generates the corresponding standardized log record object. The log dispatcher passes the log recording object to its associated log processor and calls the log filter to filter the log recording object according to the preset filtering rules. When a log recording object passes the preset filtering rules, the thread dispatcher distributes the log recording object to its corresponding file processor based on the thread identifier information carried in the log recording object; The log formatter is invoked by the file processor to format the log record object, generate log output content that conforms to a unified format template, and write the log output content to its corresponding log file.

[0068] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0069] The memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0070] As an example, the memory 406 described above may include, but is not limited to, the interception module 302, filtering module 304, distribution module 306, and output module 308 of the multi-threaded log interception and processing device. Furthermore, it may include, but is not limited to, other module units in the multi-threaded log interception and processing device described above, which will not be elaborated upon in this example.

[0071] The processor mentioned above can be a general-purpose processor, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0072] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0073] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. The device implementing the above-mentioned multi-threaded log interception and processing method can be a terminal device, such as a smartphone (e.g., Android phone, iOS phone), tablet computer, PDA, mobile Internet Devices (MID), PAD, etc. Figure 4 This does not limit the structure of the aforementioned electronic devices. For example, the electronic device may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0074] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0075] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is executed by a processor to perform the steps in the above-described method for intercepting and processing multi-threaded logs.

[0076] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0077] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0078] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0079] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for intercepting and processing multi-threaded logs, characterized in that, include: During the execution of the algorithm task, the log information generated by the standard output, uncaught exceptions, log module and third-party log library is uniformly intercepted, and the intercepted log information is sent to the log distribution recorder, which generates the corresponding standardized log record object. The log dispatcher passes the log recording object to its associated log processor and calls the log filter to filter the log recording object according to preset filtering rules. If the log recording object passes the preset filtering rules, the thread dispatch processor distributes the log recording object to its corresponding file processor according to the thread identifier information carried in the log recording object; The file processor calls the log formatter to format the log record object, generating log output content that conforms to a unified format template, and then writes the log output content into its corresponding log file.

2. The method according to claim 1, characterized in that, Before uniformly intercepting log information generated by standard output, uncaught exceptions, the logging module, and third-party logging libraries, the method also includes: When the algorithm task is executed for the first time and log interception is started, a global initialization operation is performed, wherein the global initialization operation is performed only once during the entire process lifecycle; After the global initialization operation is completed, when a new thread is detected to generate log output for the first time, the log instance initialization operation corresponding to that thread is executed.

3. The method according to claim 2, characterized in that, Performing global initialization operations includes: Obtain or create a log distribution logger, and configure the log level for the log distribution logger; Create a thread dispatch processor associated with the log dispatch logger, and register the thread dispatch processor as a log processor of the log dispatch logger; Create log filters and log formatters, and configure the log filters and log formatters to the thread dispatch processor to form a complete log processing pipeline.

4. The method according to claim 2, characterized in that, The initialization operations for the log instance corresponding to this thread include: Store the log file path or log identifier information corresponding to the algorithm task into the thread context variable; Check whether the global initialization operation has been completed, and if it is detected that the global initialization operation has not been completed, trigger the execution flow of the global initialization operation; If the global initialization operation is detected to be complete, only the initialization and storage operations of thread-local data are performed.

5. The method according to claim 1, characterized in that, The intercepted log information is sent to the log distribution recorder, which generates a corresponding standardized log recording object, including: The intercepted log information is initially parsed to extract the source of the log information, wherein the source of the log information includes the log level, timestamp, thread identifier, source file path, line number and log message content; Based on the source of the log information, the log levels are mapped or standardized to generate corresponding standardized log record objects.

6. The method according to claim 1, characterized in that, Invoking the log filter and filtering the log record objects according to preset filtering rules includes: Obtain the preset filtering rules, wherein the preset filtering rules include an exclusion path list, an exclusion message list, an inclusion path list, and an inclusion message list; Extract the message content, source file path, and related context information from the log recording object, and convert the message content and source file path of the log recording object into the corresponding comparison form according to the preset case sensitivity configuration; If the source file path of the log object begins with any of the excluded paths in the excluded path list, or the message content of the log object contains any of the excluded keywords in the excluded message list, or the source file path of the log object does not begin with any of the included paths in the included path list, or the message content of the log object does not contain any of the included keywords in the included message list, then the log object is determined to have failed the preset filtering rule and is removed from the log processing pipeline.

7. The method according to any one of claims 1 to 6, characterized in that, The file processor invokes the log formatter to format the log record object, generating log output content that conforms to a unified format template, including: Extract the source of log information from the log record object; According to the unified format template, the source of the log information is formatted and arranged, and the fixed text defined in the unified format template is merged with the variable information in the log record object to generate the corresponding log output content.

8. A device for intercepting and processing multi-threaded logs, characterized in that, include: The interception module is used to uniformly intercept log information generated by standard output, uncaught exceptions, logging module and third-party logging library during the execution of algorithm tasks, and send the intercepted log information to the log distribution recorder, which generates corresponding standardized log record objects. The filtering module is used to pass the log recording object to the associated log processor through the log distribution recorder, and call the log filter to filter the log recording object according to the preset filtering rules; The distribution module is used to distribute the log recording object to its corresponding file processor by means of the thread distribution processor according to the thread identifier information carried in the log recording object when the log recording object passes the preset filtering rules; The output module is used to call the log formatter through the file processor to format the log record object, generate log output content that conforms to a unified format template, and write the log output content to its corresponding log file.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.

Images