Ultralow-delay asynchronous formatting log processing method and system

Abstract

The invention provides an ultra-low delay asynchronous formatting log processing method and system. The method comprises the steps that a log output function is called through a producer thread; judging that the log meets the initial information of the standard format based on the macro expansion operation; the data type corresponding to the initial information comprises a non-character string type and a character string type; executing different storage strategies according to the data type so as to store the initial log information into a TLS lock-free buffer area corresponding to a producer thread; the method comprises the following steps: acquiring initial log information from a TLS lock-free buffer area by a consumer thread; performing asynchronous formatting processing on the initial log information to obtain target log information; and finally, writing the target log information into the log file. The log output efficiency can be improved, and ultra-low time delay of log recording is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a super-low-delay asynchronous formatted log processing method and system. BACKGROUND

[0002] In the existing software system development process, a log system is an important component indispensable, and is widely applied to debugging, monitoring and fault troubleshooting scenes. At present, most of the common log libraries are based on glibc's fprintf (such as vsnprintf) series of standard library functions, or use high-performance log libraries such as spdlog.

[0003] Although the existing log library has basic log formatting and output functions, it generally has the following deficiencies: (1) high time consumption of synchronous formatting: most log libraries execute string formatting in a business thread (a thread sensitive to delay), and this step involves complex string splicing and memory operation, which leads to high time consumption of the business thread in recording logs, and is easy to become a performance bottleneck. (2) I / O blocking problem: many log libraries directly execute I / O operations of writing files or devices in the business thread, which is easy to cause the business thread to be blocked, and affects the real-time performance and throughput of the main business logic. (3) serious resource competition in a multi-threaded environment: some log systems use spin locks and other ways to synchronize when writing logs in a multi-threaded environment, which is easy to cause lock competition problems, and the performance is obviously deteriorated in a high-concurrency scene.

[0004] The existing log system has defects such as performance bottleneck and serious blocking problem, and there is an urgent need for an efficient asynchronous log system that can balance performance and versatility and achieve extremely low delay in the business thread. SUMMARY

[0005] The present application provides a super-low-delay asynchronous formatted log processing method and system, which can accurately record real-time log information according to a user-defined format string by calling a log output function by a producer thread, and guarantees the accuracy of the log through format checking at the compilation stage. Different storage strategies are executed according to the data type to store the log into a TLS lock-free buffer, which avoids thread competition and improves storage efficiency. The consumer thread asynchronously formats and writes the log into a file, so that the producer thread does not need to wait, which significantly reduces the impact of log recording on the business, and realizes super-low-delay and efficient operation of log processing.

[0006] In a first aspect, the present application provides a super-low-delay asynchronous formatted log processing system, which comprises: The first processing module is configured to call a log output function by a producer thread, the log output function including a format string and a parameter list, the format string being used to define a standard format of log information, and the parameter list containing initial log information generated by the producer thread in real time; and based on a macro expansion operation, it is determined that the initial log information meets the standard format; and a data type corresponding to the initial log information is determined, the data type including a non-string type and a string type; and different storage strategies are performed according to the data type, so as to store the initial log information into a TLS lock-free buffer corresponding to the producer thread. The second processing module is configured to detect the initial log information from the TLS lock-free buffer by a consumer thread; perform asynchronous formatting processing on the initial log information to obtain target log information; and write the target log information into a log file.

[0007] In a second aspect, an embodiment of the present application provides an ultra-low latency asynchronous formatted log processing method, applied to an ultra-low latency asynchronous formatted log processing system, and the method includes: A log output function is called by a producer thread, the log output function including a format string and a parameter list, the format string being used to define a standard format of log information, and the parameter list containing initial log information generated by the producer thread in real time; and based on a macro expansion operation, it is determined that the initial log information meets the standard format. A data type corresponding to the initial log information is determined, the data type including a non-string type and a string type. Different storage strategies are performed according to the data type, so as to store the initial log information into a TLS lock-free buffer corresponding to the producer thread. The initial log information is detected from the TLS lock-free buffer by a consumer thread. Asynchronous formatting processing is performed on the initial log information to obtain target log information. The target log information is written into a log file.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a memory, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the processor, and the program includes instructions for performing steps in the first aspect of the embodiments of the present application.

[0009] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the steps of the method in the first aspect.

[0010] It can be seen that, in the embodiments of the present application, the producer thread calls the log output function; and it is determined that the initial log information meets the standard format based on the macro expansion operation; the data type corresponding to the initial log information, the data type including a non-string type and a string type; different storage strategies are executed according to the data type to store the initial log information into the TLS lock-free buffer corresponding to the producer thread; the initial log information is detected from the TLS lock-free buffer by the consumer thread; the initial log information is subjected to asynchronous formatting processing to obtain target log information; and the target log information is written into a log file. In this way, compared with the existing log formatting processing scheme, the present scheme adopts the producer-consumer asynchronous mode, and transfers the formatting and I / O operation to the background thread to avoid the efficiency reduction of the business thread due to the synchronous formatting and I / O blocking; the TLS lock-free is used to eliminate the resource competition of multiple threads and reduce the lock overhead; meanwhile, the log processing accuracy and efficiency are improved by the compile-time format checking and the targeted storage strategy, the ultra-low latency of log recording is realized, and the real-time performance and throughput of the system in the high concurrency scenario are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0012] Figure 1 is a structural schematic diagram of an ultra-low latency asynchronous formatting log processing system provided by the embodiments of the present application; Figure 2 is a step flowchart of an ultra-low latency asynchronous formatting log processing method provided by the embodiments of the present application; Figure 3 is a flowchart of a first storage strategy provided by the embodiments of the present application; Figure 4 is a flowchart of a second storage strategy provided by the embodiments of the present application; Figure 5 is a flowchart of log information asynchronous formatting processing provided by the embodiments of the present application; Figure 6is a whole flowchart of a method for processing an ultra-low latency asynchronous formatted log provided by an embodiment of the present application. Figure 7 is a structural block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0013] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work fall within the scope of protection of the present application.

[0014] The terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product, or device.

[0015] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0016] In the embodiments of the present application, "and / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships. For example, A and / or B can represent the following three cases: A exists alone; A and B exist simultaneously; B exists alone. Wherein, A and B can be singular or plural.

[0017] In the embodiments of the present application, the symbol " / " can represent that the associated objects before and after it are in an "or" relationship. In addition, the symbol " / " can also represent the division sign, that is, to perform division operation. For example, A / B can represent A divided by B.

[0018] In the embodiments of the present application, "at least one item" or similar expressions refers to any combination of these items, including any combination of single items or plural items, and refers to one or more, and multiple refers to two or more. For example, at least one item (item) of a, b, or c can represent the following seven situations: a, b, c, a and b, a and c, b and c, a, b, and c. Among them, each of a, b, and c can be an element or a set containing one or more elements.

[0019] In the embodiments of this application, "equal to" can be used in conjunction with "greater than" and is applicable to the technical solution adopted when "greater than" is used, and can also be used in conjunction with "less than" and is applicable to the technical solution adopted when "less than" is used. When "equal to" is used in conjunction with "greater than", it should not be used in conjunction with "less than"; when "equal to" is used in conjunction with "less than", it should not be used in conjunction with "greater than".

[0020] In order to better understand the solutions of the embodiments of the present application, the electronic devices, related concepts and backgrounds that may be involved in the embodiments of the present application are first introduced below.

[0021] (1) Business threads (producer threads): In scenarios with extremely high performance requirements, such as real-time transactions, these threads are responsible for key business logic processing and computing tasks. They are directly involved in the operation of core business processes. Their operating efficiency and response speed play a decisive role in the real-time and accuracy of the overall business. Because these threads are extremely sensitive to time delays, any unnecessary time-consuming operations may affect the normal operation of the business. Therefore, it is necessary to minimize blocking and waiting time.

[0022] (2) Consumer thread: In contrast to the business thread, it does not participate in the calculation process of the key business path and is mainly responsible for processing the subsequent operations of the log data generated by the business thread. This thread focuses on formatting the temporarily stored log data and writing it to the storage medium (such as log files). Because it does not involve core business functions, it is less sensitive to time delays and can be flexibly scheduled and executed in the background to avoid affecting the operation of key business threads.

[0023] (3) Formatting: Based on the formatting string template pre-set by the user, the original log data is converted into a readable and standardized text format according to specific rules. Through this conversion, the log information can be presented in a unified and structured format.

[0024] (4) Synchronous formatting: When a business thread calls a log output function, it immediately formats the log data within the current thread. During this process, the business thread must wait for the formatting task to complete before continuing with subsequent work. If the formatting process is complex (for example, involving a large number of string concatenation, format conversion, and other operations), it may cause significant delays, thereby affecting the execution efficiency of the business thread and even becoming a performance bottleneck for the entire system.

[0025] (5) Asynchronous formatting: After the business thread calls the log output function, it only completes the temporary storage of data within the current thread quickly, without formatting processing, but hands over the formatting task to other threads (such as consumer threads) for execution. This way avoids the time-consuming and blocking of the business thread due to formatting operations, ensures the smoothness of the key business process, enables the business thread to focus on core business processing, and significantly improves the overall response speed and running efficiency of the system.

[0026] (6) TLS lock-free buffer: Full name is thread-local storage lock-free buffer, which is a high-efficiency data storage structure applied in a multi-threaded environment. Based on thread-local storage (TLS) technology, it allocates independent cache space for each thread, avoiding the competition of multiple threads for shared resources. With the lock-free design, it uses atomic operations to ensure the atomicity and consistency of data access and modification, avoiding the thread blocking and context switching overhead caused by the traditional lock mechanism, and greatly improving the data read-write efficiency.

[0027] (7) rdtsc instruction: It is an assembly instruction in x86 architecture processor, full name is "Read Time-Stamp Counter". It is used to read the value of the time stamp counter in the processor, which will be incremented with the clock cycle of the processor. Through the rdtsc instruction, the program can accurately obtain the relative time of instruction execution, and its precision can reach nanosecond level.

[0028] Although the existing log library has basic log formatting and output functions, it generally has the following shortcomings: (1) high time-consuming of synchronous formatting: most log libraries perform string formatting in business threads (threads sensitive to latency), which involves complex string concatenation and memory operations, resulting in high time-consuming of business threads recording logs, which is easy to become a performance bottleneck. (2) I / O blocking problem: many log libraries directly perform I / O operations such as writing files or devices in business threads, which can easily block business threads and affect the real-time performance and throughput of the main business logic. (3) Serious resource competition in multi-threaded environment: some log systems use spin locks and other synchronization methods when writing logs in multi-threaded concurrent mode, which can easily cause lock competition problems, especially in high-concurrency scenarios.

[0029] In view of the above problems, the embodiment of the present application provides a super low latency asynchronous formatting log processing method and system, which will be described in detail below in combination with the drawings.

[0030] Please refer to Figure 1 , Figure 1 is a structural schematic diagram of a super low latency asynchronous formatting log processing system provided by the embodiment of the present application, as Figure 1As shown, the ultra-low latency asynchronous formatted log processing system comprises the following modules: The first processing module 110 is configured to call a log output function by a producer thread, the log output function comprising a format string and a parameter list, the format string being used to define a standard format of log information, the parameter list containing initial log information generated by the producer thread in real time; and determining, based on a macro expansion operation, that the initial log information meets the standard format; and determining a data type corresponding to the initial log information, the data type comprising a non-string type and a string type; and performing different storage strategies according to the data type to store the initial log information into a TLS lock-free buffer corresponding to the producer thread.

[0031] The log output function contains two core elements, a format string and a parameter list. The format string adopts a specific syntax structure and is used to define a standard output format of log information, covering data type placeholders, text separators and format modifiers, etc., thereby providing a rule basis for the formatting of log information. The parameter list carries initial log data generated by the producer thread in real time, including but not limited to business event parameters, system state variables and other information.

[0032] In one possible embodiment, the system further comprises a format definition module, wherein: The format definition module is configured to receive a standard format of log information input by a user before the log output function is called by the producer thread, and construct the format string according to the standard format of log information.

[0033] For example, the log output function can be represented as myinfox (format string fmt, parameter list args…). fmt is the format string, which defines the basic structure and style of log information. The string can contain ordinary text and placeholders, which reserve positions for subsequent insertion of actual data. The form of the placeholder varies with programming languages and specific log libraries. The parameter list args contains actual data to replace the placeholders in fmt. The order and type of these parameters need to match the order and type of the placeholders in fmt.

[0034] Specifically, the user calls the function, inputs related parameters such as strings, integers, floating-point types, etc. and a format string to explicitly define the standard format of log information. Then, the function internally performs a series of operations of the producer thread (checking parameter types, counting parameter numbers, calculating offsets, recording timestamps, writing into a TLS lock-free buffer, etc.) and the consumer thread (checking new logs, processing time, formatting logs, writing into files) in this solution to write log information into a log file.

[0035] The macro expansion operation is mainly used to perform format checking on the initial log information at the compilation stage to ensure that it meets the standard format defined by the format string. Specifically, the compiler will expand the macro and perform detailed comparison between the format string and the parameter list. For example, it checks whether the format specifiers in the format string (such as %d for integer and %s for string) are consistent with the data types in the parameter list.

[0036] In one possible embodiment, the first processing module is further configured to output a warning information when it is determined based on the macro expansion operation that the initial log information does not meet the standard format, the warning information being used to prompt a user to modify the standard format.

[0037] In a specific implementation, the macro expansion operation is performed at the compilation stage, which means that the format checking is completed before the code is run. The compiler will perform detailed analysis and comparison between the format string and the parameter list. When it is determined that the initial log information does not meet the standard format, the system will output the warning information to a suitable location, such as the output window of the compiler, a log file, or a specific error prompt interface. The warning information usually explicitly indicates the specific location of the format mismatch and the problem, for example, "the format string %s expects a string parameter, but a non-string type is passed in", so that the user can quickly understand the problem and make modifications.

[0038] As can be seen, in this embodiment, the format checking is performed at the compilation stage and the warning information is output, which can discover potential format errors before the code is run, avoiding problems that are difficult to debug at runtime. This helps to improve development efficiency and reduce system failures and downtime caused by format errors.

[0039] In one possible embodiment, the first processing module includes: a storage area determination unit configured to determine the number of initial log information in the parameter list by a VA_NARG macro operation, and to apply for a target cache area from the TLS lock-free buffer according to the number of parameters of the initial log information; a time recording unit configured to call an rdtsc instruction to record a timestamp; a type determination unit configured to determine the data type corresponding to the initial log information; a storage unit configured to, when the data type is a non-string type, execute a first storage strategy to store the initial log information and the timestamp in the target cache area, and configured to, when the data type is a string type, execute a second storage strategy to store the target address data corresponding to the initial log information and the timestamp in the target cache area.

[0040] VA_NARG macro operation is used to determine the number of parameters in the initial log information parameter list. In C or C++ programming, variable parameter functions or macros often need to know the number of parameters for subsequent processing. The VA_NARG macro is a custom macro whose implementation can count the number of variable parameters by some compiler features or specific syntax rules. For example, it may analyze the length of the parameter list or use specific markers to identify the end of the parameter. Through this macro operation, the system can accurately understand how many initial log information needs to be processed.

[0041] It can be understood that different amounts of log information require different storage space, and according to the number of parameters of the log information, the cache area can be applied to ensure that the size of the cache area applied is appropriate, neither too large to cause resource waste, nor too small to cause log information storage.

[0042] In specific implementation, in addition to the number of parameters of the log information, the data type and size also affect the application of the cache area. For example, non-string type data usually occupies fixed storage space, while string type data length may be different, and the required storage space will also be different. When applying for a cache area, these factors need to be considered comprehensively to ensure that enough storage space can be provided for different types and sizes of log information.

[0043] Further, the target cache area applied is marked, and its start address and size information are recorded, so that subsequent storage operations can accurately write log information into the area.

[0044] Among them, the rdtsc instruction is used to read the value of the processor's internal timestamp counter (TSC), which will continuously increase with the processor's clock period. In the log processing system, the rdtsc instruction can accurately record the time when the log information is generated. Since it is a hardware-level operation, the execution speed is very fast, and it can obtain high-precision timestamps in a very short time, providing accurate time basis for subsequent log analysis and sorting.

[0045] In one possible embodiment, the first storage strategy is to calculate the first offset of the initial log information, and store the initial log information and the timestamp into the target cache area according to the first offset; The second storage strategy is to copy the initial log information to a designated area in the TLS lock-free buffer through an lmark macro operation, determine the target address data of the initial log information in the designated area, calculate a second offset of the target address data, and store the target address data and the timestamp in the target cache area according to the second offset.

[0046] Wherein, the offset is a relative position concept, in computer memory management and data storage, it represents the distance from a fixed starting address to the target data location, usually in bytes.

[0047] Specifically, the calculation of the offset is usually based on the starting address of the target cache area, the size of the stored data, and the size of the non-string type data itself. By accurately calculating the offset, it can ensure that the data is stored in the cache area in a certain order, avoiding data overlap or confusion.

[0048] Wherein, the lmark macro operation is specially designed for processing initial log information of string type. In the ultra-low latency asynchronous formatted log processing system, the length of string data is not the same, in order to efficiently store and manage these data, the lmark macro operation is introduced. One of the main functions of lmark macro operation is to copy the initial log information of string type to the designated area in the TLS lock-free buffer. It may use some optimization methods, such as memory alignment, batch copy, etc., to improve the efficiency of copying.

[0049] Further, after completing the string copy, the lmark macro operation will determine the target address data of the string in the designated area. This address data serves as a reference for the string, and only the address needs to be saved during subsequent storage, rather than the entire string content, saving space in the target cache area.

[0050] In one possible embodiment, the storage unit is further configured to, when the initial log information simultaneously includes non-string type information and string type information, execute a third storage strategy to store the non-string type information, the target address data corresponding to the string type information, and the timestamp in the target cache area.

[0051] In a possible embodiment, the third storage strategy comprises: calculating a third offset of the non-string type information; copying the string type information to a specified area in the TLS lock-free buffer through an lmark macro operation; determining the target address data of the string type information in the specified area; calculating a fourth offset of the target address data; and storing the target address data corresponding to the non-string type information and the string type information into the target cache area according to the third offset and the fourth offset; and storing the timestamp into the target cache area.

[0052] In a possible embodiment, the storing the target address data corresponding to the non-string type information and the string type information into the target cache area according to the third offset and the fourth offset comprises: determining a start address of the target cache area; offsetting a corresponding number of bytes from the start address of the target cache area according to the third offset to obtain a first storage position of the non-string type information, and writing the non-string type information into the first storage position; offsetting a corresponding number of bytes from the start address of the target cache area according to the fourth offset to determine a second storage position of the target address data corresponding to the string type information, and writing the target address data into the second storage position.

[0053] Exemplarily, the start address of the target cache area is 0x1000, the non-string type information is 123, the third offset is 0, the string type information is "Hello", the string is copied to a specified area in the TLS lock-free buffer through an lmark macro operation, the target address data of the string is 0x2000, and the fourth offset is 4. Since the third offset is 0, the non-string type information 123 is stored at the start address 0x1000 of the target cache area. In the memory, 0x1000-0x1003 are four bytes storing the binary representation of the integer 123. Since the fourth offset is 4, 0x1004 is obtained by offsetting 4 bytes from the start address 0x1000 of the target cache area. The string target address data 0x2000 is stored in 0x1004-0x1007.

[0054] In a possible embodiment, the first processing module further comprises: an updating unit configured to update a latest usage state of the TLS lock-free buffer, the latest usage state being used to represent whether the initial log information is newly stored in the target cache area.

[0055] The latest usage state is used to represent whether the initial log information is newly stored in the target cache area, which includes but is not limited to being implemented by a Boolean type identifier, a state enumeration, a counter, or a timestamp. By updating and checking the latest usage state, the consumer thread can avoid unnecessary polling and empty operations. Only when the latest usage state indicates that there is new log information stored, the consumer thread will read and process the log from the buffer, thereby reducing the waste of system resources and improving the efficiency of log processing.

[0056] It can be seen that, in the embodiment, the log output function and the macro expansion operation implement accurate log format verification and error warning at the compilation stage, effectively avoiding the risk of failure caused by runtime format errors; with the help of VA_NARG macro, rdtsc instruction, lmark macro and other technologies, combined with storage strategies and offset calculations for different data types, efficient and orderly storage of log information is achieved, ensuring reasonable use of storage space and high-precision recording of time stamps; at the same time, the dynamic management of the latest usage state of the TLS lock-free buffer by the updating unit avoids invalid polling of the consumer thread, significantly improves the log processing efficiency, reduces the system resource consumption, and finally achieves low latency, high reliability and high performance of log processing.

[0057] The second processing module 120 is configured to detect the initial log information from the TLS lock-free buffer by a consumer thread; perform asynchronous formatting processing on the initial log information to obtain target log information; and write the target log information into a log file.

[0058] The consumer thread and the producer thread run on different CPU cores, which is a multi-thread parallel processing mode. By allocating different tasks to different CPU cores, the performance advantages of multi-core processors can be fully utilized to improve the overall processing capacity of the system.

[0059] Further, when the log information processed by multiple consumer threads has certain correlation or locality, binding them to one CPU core can effectively utilize CPU resources and avoid frequent switching between multiple threads on different cores. Data sharing and communication between threads are more efficient, reducing the delay of cross-core data transmission.

[0060] Among them, the CPU core is an independent processing unit in the central processing unit (CPU), which can independently execute instructions and process data. Modern multi-core processors usually contain multiple CPU cores, and each core can run one or more threads simultaneously. By allocating different tasks to different CPU cores, parallel processing can be achieved, improving the overall performance of the computer system.

[0061] In one possible embodiment, the second processing module comprises: The detection unit is configured to determine whether the initial log information is newly stored in the target cache area according to the latest usage state of the TLS lock-free buffer area. The prefix adding unit is configured to restore the microsecond time from the timestamp recorded by the rdtsc instruction when it is detected that the initial log information is newly stored in the target cache area, and add a time prefix to the initial log information according to the microsecond time. The format processing unit is configured to perform format processing on the initial log information according to the format string based on the vsnprintf function, so that the placeholder can be replaced according to the structure of the format string to generate the target log information, and the format string contains the placeholder. The log file unit is configured to write the target log information into the log file.

[0062] Among them, the vsnprintf function is a function in the C language standard library, which can output formatted data to a character array according to the given format string and parameter list.

[0063] In one possible embodiment, the prefix adding unit is further configured to give up the time slice when it is detected that the initial log information is not newly stored in the target cache area.

[0064] Among them, the time slice refers to a fixed time interval allocated by the CPU to each running thread. By giving up the time slice, the CPU can execute other threads that have tasks to be processed. When new logs arrive, the consumer thread will obtain the time slice again and continue to execute the subsequent log processing task.

[0065] It can be seen that, in the embodiment, the producer thread calls the log output function, and the real-time log information can be accurately recorded according to the format string customized by the user. The log is stored in the TLS lock-free buffer according to different storage strategies of different data types, which avoids thread competition and improves storage efficiency. The consumer thread asynchronously formats and writes the log into the file, so that the producer thread does not need to wait, which significantly reduces the impact of log recording on business, and realizes ultra-low latency and efficient operation of log processing.

[0066] Please refer to Figure 2 , Figure 2 is a step flow chart of an ultra-low latency asynchronous formatting log processing method provided by the embodiment of the application, as shown in Figure 2 , the method comprises the following steps: Step S201, calling a log output function by a producer thread.

[0067] The log output function comprises a format string and a parameter list, the format string is used to define the standard format of log information, and the parameter list comprises initial log information generated by the producer thread in real time.

[0068] In one possible embodiment, the method further comprises: before the log output function is called by the producer thread, receiving the standard format of the log information input by the user, and constructing the format string according to the standard format of the log information.

[0069] For example, the log output function can be represented as myinfox (format string fmt, parameter list args…). Fmt is a format string, which defines the basic structure and style of the log information. The string can contain ordinary text and placeholders, and the placeholders are used to reserve positions for subsequent insertion of actual data. The form of the placeholder will be different due to the programming language and the specific log library.

[0070] Step S202, judging that the initial log information meets the standard format based on a macro expansion operation.

[0071] The macro expansion operation is mainly used to perform format checking on the initial log information at the compilation stage, to ensure that it meets the standard format defined by the format string. Specifically, the compiler will expand the macro, and compare the format string and the parameter list in detail.

[0072] In one possible embodiment, the method further comprises: when it is judged that the initial log information does not meet the standard format based on the macro expansion operation, outputting warning information, the warning information is used to prompt the user to modify the standard format.

[0073] In a specific implementation, the macro expansion operation is performed during the compilation phase, which means that the format checking is completed before the code runs. The compiler will perform a detailed analysis and comparison of the format string and the parameter list, and when it is determined that the initial log information does not meet the standard format, the system will output warning information to the appropriate location, such as the compiler's output window, log file, or specific error prompt interface.

[0074] As can be seen, in this embodiment, format checking and warning information output are performed during the compilation period, which can detect potential format errors before the code runs, avoiding difficult-to-debug problems at runtime. This helps to improve development efficiency and reduce system failures and downtime caused by format errors.

[0075] Step S203, determine the data type corresponding to the initial log information.

[0076] The data type includes a non-string type and a string type.

[0077] Step S204, according to the data type, different storage strategies are executed to store the initial log information into the TLS lock-free buffer corresponding to the producer thread.

[0078] In one possible embodiment, the different storage strategies according to the data type include: determining the number of initial log information in the parameter list through the VA_NARG macro operation; and according to the number of parameters of the initial log information, a target cache area is applied from the TLS lock-free buffer; the rdtsc instruction is called to record the timestamp; when the data type is a non-string type, a first storage strategy is executed to store the initial log information and the timestamp into the target cache area; and when the data type is a string type, a second storage strategy is executed to store the target address data corresponding to the initial log information and the timestamp into the target cache area.

[0079] The VA_NARG macro operation is used to determine the number of parameters of the initial log information in the parameter list. In C or C++ programming, variable parameter functions or macros often need to know the number of parameters for subsequent processing. VA_NARG macro is a self-defined macro, and its implementation principle can be some compiler features or specific syntax rules to count the number of variable parameters.

[0080] One of the main functions of the lmark macro operation is to copy the initial log information of the string type to the specified area in the TLS lock-free buffer. It may use some optimization methods such as memory alignment, batch copying, etc. to improve the efficiency of copying.

[0081] In a possible embodiment, the method further includes: updating a latest usage state of the TLS lock-free buffer, the latest usage state being used to represent whether the initial log information is newly stored in the target cache area.

[0082] The latest usage state is used to represent whether the initial log information is newly stored in the target cache area, and is implemented by, but not limited to, a Boolean type identifier, a state enumeration, a counter, or a timestamp. By updating and checking the latest usage state, the consumer thread can avoid unnecessary polling and empty operations.

[0083] Step S205: detecting, by the consumer thread, the initial log information from the TLS lock-free buffer.

[0084] The consumer thread and the producer thread run on different CPU cores, which is a multi-thread parallel processing mode. When the log information processed by multiple consumer threads has certain correlation or locality, binding them to one CPU core can effectively utilize CPU resources and avoid frequent switching between multiple threads on different cores. Data sharing and communication between threads are more efficient, and the delay of cross-core data transmission is reduced.

[0085] In a possible embodiment, the method further includes: yielding a time slice when it is detected that the initial log information is not newly stored in the target cache area.

[0086] The time slice refers to a fixed time interval allocated by the CPU to each running thread. By yielding the time slice, the CPU can execute other threads that have tasks to be processed. When new logs arrive, the consumer thread will obtain the time slice again and continue to execute subsequent log processing tasks.

[0087] Step S206: performing asynchronous formatting processing on the initial log information to obtain target log information.

[0088] Before the formatting processing, the microsecond-level time is restored according to the timestamp recorded by the rdtsc instruction, and a time prefix is added to the initial log information according to the microsecond-level time.

[0089] Further, the initial log information is formatted according to the format string based on the vsnprintf function.

[0090] Step S207: writing the target log information into a log file.

[0091] It can be seen that, in the embodiment, the producer thread calls the log output function, and the real-time log information can be accurately recorded according to the user-defined format string. The log accuracy is ensured through the format checking at the compiling stage. The log is stored in the TLS lock-free buffer according to different storage strategies of data types, so as to avoid thread competition and improve storage efficiency. The consumer thread asynchronously formats and writes the log into a file, so that the producer thread does not need to wait, the influence of log recording on business is significantly reduced, and the ultra-low latency and efficient operation of log processing are realized.

[0092] Please refer to Figure 3 , Figure 3 is a flowchart of a first storage strategy provided by the embodiment of the application, wherein when the data type is a non-string type, the first storage strategy is executed, and the method can further include the following steps: Step S301, it is judged that the data type of initial log information is a non-string type.

[0093] Step S302, an rdtsc instruction is called to record a timestamp.

[0094] The rdtsc instruction is used to read the value of a timestamp counter (TSC) in the processor, and the counter is continuously incremented with the clock cycle of the processor.

[0095] Step S303, a first offset of the initial log information is calculated.

[0096] The offset is a concept of relative position, and in computer memory management and data storage, it represents the distance from a fixed starting address to the position of target data, usually in bytes. By accurately calculating the offset, the data can be stored in the cache area in a certain order, avoiding data overlap or confusion.

[0097] Step S304, the initial log information and the timestamp are stored in a target cache area according to the first offset.

[0098] The target cache area is located in the TLS lock-free buffer, and the determination method of the target cache area specifically includes: determining the number of initial log information in the parameter list through a VA_NARG macro operation; and applying the target cache area from the TLS lock-free buffer according to the parameter number of the initial log information.

[0099] VA_NARG macro operation is used to determine the number of parameters of the initial log information in the parameter list. In C or C++ programming, variable parameter functions or macros often need to know the number of parameters for subsequent processing. The VA_NARG macro is a self-defined macro, and its implementation principle can be to count the number of variable parameters by some compiler features or specific syntax rules.

[0100] In a specific implementation, in addition to the number of parameters of the log information, the data type and size also affect the application of the cache area. For example, non-string type data usually occupies a fixed storage space, while the length of string type data may be different, and the required storage space will also be different. When applying for a cache area, these factors need to be considered comprehensively to ensure that enough storage space can be provided for log information of different types and sizes.

[0101] As can be seen, in the embodiment, the first storage strategy is for non-string type initial log information, and the rdtsc instruction is called to accurately record the timestamp, the VA_NARG macro operation is used to determine the number of information, and the target cache area is applied from the TLS lock-free buffer according to the number of information, and the first offset is calculated to realize the ordered storage of data in the cache area. This strategy effectively avoids data overlap and confusion, ensures efficient and accurate storage of different non-string type log information, improves log storage efficiency and reliability, and provides an ordered and standardized data basis for subsequent log processing.

[0102] Referring to Figure 4 , Figure 4 is a flowchart of a second storage strategy provided by the embodiment of the application, wherein when the data type is a string type, the second storage strategy is executed, and the above method can further include the following steps: Step S401, it is judged that the data type of the initial log information is a string type.

[0103] Step S402, the rdtsc instruction is called to record the timestamp.

[0104] The rdtsc instruction is used to read the value of the time stamp counter (TSC) in the processor, and the counter will be incremented with the clock cycle of the processor.

[0105] Step S403, the initial log information is copied to the specified area in the TLS lock-free buffer by the lmark macro operation.

[0106] The lmark macro operation is specially designed for processing the initial log information of the string type. In the ultra-low latency asynchronous log formatting processing system, the lengths of the string data are different. In order to efficiently store and manage the data, the lmark macro operation is introduced. One of the main functions of the lmark macro operation is to copy the initial log information of the string type to a specified area in the TLS lock-free buffer. It may use some optimization methods such as memory alignment, batch copying and the like to improve the efficiency of copying.

[0107] In step S404, the target address data of the initial log information in the specified area is determined.

[0108] In the specific implementation, after the string copying is completed, the lmark macro operation determines the target address data of the string in the specified area. The address data serves as a reference of the string, and the address is saved instead of the entire string content in subsequent storage, thereby saving the space of the target cache area.

[0109] In step S405, a second offset of the target address data is calculated.

[0110] In step S406, the target address data and the timestamp are stored in the target cache area according to the second offset.

[0111] The target cache area is located in the TLS lock-free buffer, and the determination method of the target cache area specifically includes: determining the number of the initial log information in the parameter list through the VA_NARG macro operation; and applying the target cache area from the TLS lock-free buffer according to the parameter number of the initial log information.

[0112] It can be seen that, in the embodiment, for the initial log information of the string type, the second storage strategy accurately records the timestamp through the rdtsc instruction, realizes efficient copying and address reference of the string by means of the lmark macro operation, avoids the space waste caused by directly storing the long string, accurately positions the data storage position by calculating the second offset, cooperates with the VA_NARG macro operation to determine the target cache area, realizes efficient storage and ordered management of the string log information, reduces the memory occupation, guarantees the accuracy and efficiency of the log storage, and provides a reliable data basis for subsequent log processing.

[0113] Please refer to Figure 5 , Figure 5 is a flowchart of a log information asynchronous formatting processing provided by the embodiment of the present application, wherein the log information asynchronous formatting is executed by a consumer thread, and the method can further include the following steps. Step S501, judging whether the initial log information is newly stored in the target cache area according to the latest usage state of the TLS lock-free buffer.

[0114] The latest usage state is used to represent whether the initial log information is newly stored in the target cache area, including but not limited to being realized by a Boolean type identifier, a state enumeration, a counter or a timestamp. By updating and checking the latest usage state, the consumer thread can avoid unnecessary polling and empty operations.

[0115] Specifically, if yes, step S502 is executed; and if no, step S503 is executed. Step S502, restoring the microsecond-level time according to the timestamp recorded by the rdtsc instruction.

[0116] It can be understood that the rdtsc instruction reads the value of the processor internal timestamp counter (TSC), which is a counting value that increases with the processor clock cycle, and is not an actual time unit (such as seconds, microseconds). To restore it to the microsecond-level time, the processor clock frequency needs to be converted.

[0117] Step S503, yielding the time slice.

[0118] The time slice refers to a fixed time interval allocated by the CPU to each running thread. By yielding the time slice, the CPU can execute other threads that have tasks to be processed. When new logs arrive, the consumer thread will obtain the time slice again and continue to execute the subsequent log processing task.

[0119] Step S504, adding a time prefix to the initial log information according to the microsecond-level time.

[0120] In specific implementation, after obtaining the microsecond-level time, the time information is converted into a string form according to a certain format, and then added to the beginning of the initial log information to form a time prefix. For example, a prefix like "2025-04-24 15:30:00.123456" is generated, which is spliced with the original log content to form complete log information with time identification.

[0121] Step S505, based on the vsnprintf function, the initial log information is formatted according to the format string to generate the target log information.

[0122] The vsnprintf function is a function in the C language standard library, which outputs the formatted data to a character array according to the given format string and parameter list.

[0123] Step S506, write the target log information into a log file.

[0124] It can be seen that, in the embodiment, whether there is new log to be stored is determined by checking the latest use state of the TLS lock-free buffer, so as to avoid invalid polling. If there is new log, the timestamp recorded by the rdtsc instruction is converted into microsecond level and added to the log information as a prefix, then the target log information is generated by formatting using the vsnprintf function, and finally written into the log file; if there is no new log, the time slice is given up. This process accurately and efficiently processes the log, improves the CPU utilization, guarantees the timeliness and standardization of log recording, and provides strong support for subsequent log analysis.

[0125] Please refer to Figure 6 , Figure 6 is a whole flowchart of an ultra-low latency asynchronous formatted log processing method provided by the embodiment of the present application, as shown in Figure 6 , the method comprises: calling a log output function (format string fmt, parameter list args…) by a producer thread; checking whether the format of initial log information in the format string and the parameter list matches in the compilation period, otherwise warning; counting the actual number of initial log information; traversing each initial log information and calculating the offset; and, in the running time, applying a target cache area from the TLS lock-free buffer; recording the timestamp by the rdtsc instruction; sequentially writing a plurality of initial log information into the target cache area corresponding to the offset; updating the latest use state of the TLS lock-free buffer; and, in the running time, checking whether there is new log in the TLS lock-free buffer by a consumer thread, otherwise giving up the time slice; adding the microsecond level time prefix according to the timestamp recorded by the rdtsc instruction; based on the vsnprintf function, formatting the initial log information according to the format string to obtain the target log information; and writing the target log information into a log file.

[0126] Among them, sequentially writing a plurality of initial log information into the target cache area corresponding to the offset, it is necessary to consider whether the data type of the initial log information is a non-string type, if yes, the first storage strategy is executed, otherwise the second storage strategy is executed, so as to store initial log information of different types into the TLS lock-free buffer.

[0127] In one possible embodiment, the first storage strategy is to calculate the first offset of the initial log information, and store the initial log information and the timestamp into the target cache area according to the first offset; The second storage strategy is to copy the initial log information to a specified area in the TLS lock-free buffer through the lmark macro operation; to determine target address data of the initial log information in the specified area; to calculate a second offset of the target address data, and to store the target address data and the timestamp into the target cache area according to the second offset.

[0128] It can be understood that the use mode of the log library provided by the scheme is designed to be similar to the printf library function, specifically written in pure C language, and the code structure is relatively simple compared to other log systems on the market, does not require a high version of gcc compiler (gcc4.x is supported), and does not require new language standard support, C / C++ can be applied.

[0129] In a specific implementation, compared with the existing log library "spdlog", the log library provided by the scheme has faster printing speed and far superior performance to the spdlog log library in printing specific format logs under different thread numbers. The printing refers to the process of storing log information containing strings, integers, floating-point types, etc. in the TLS lock-free buffer according to the specified format string, preprocessing by the producer thread, reading by the consumer thread, restoring time, adding time prefix and formatting, and writing into the log file, which is a key operation to realize log persistent storage. For details, see the following log printing performance test comparison table: Log printing performance test comparison table Specifically, the test is based on the GoogleBenchmark library and runs on a CPU with 24 cores and a frequency of 2996.46MHz. The CPU cache at each level includes L1 data cache (32KiBx24), L1 instruction cache (32KiBx24), L2 unified cache (256KiBx24), and L3 unified cache (30720KiBx2). The size and hit rate of the cache will affect the read and write speed of the data. The average load is (0.12, 0.21, 0.16), which indicates that the system load is relatively low during the test, and the possibility of interference from other tasks in the system is small, so the test results can accurately reflect the performance of each log library.

[0130] Further, the mylogger log library and the spdlog log library provided by the scheme are used to print a string, three integers, and three floating-point types in this test, and the corresponding format string is "symbol:% s time:%d last:%.2f bp1:%.2fbv1:%d ap1:%.2f av1:%d".

[0131] Specifically, numbers 1-4 show the actual time consumption and CPU time consumption of the mylogger log library provided by the present solution to print logs under different threads (threads) for multiple iterations. As the number of threads increases from 1 to 8, the actual time consumption of the mylogger log library significantly decreases from 13.8 ns to 1.83 ns, which indicates that the mylogger log library has excellent multi-thread scalability, can effectively utilize multi-core CPU resources, has strong parallel processing capability, and the CPU time consumption is relatively stable, maintaining between 13.7-14.6 ns, indicating that its utilization efficiency of CPU resources is relatively stable and does not fluctuate greatly due to the change of the number of threads.

[0132] Further, numbers 5-8 show the actual time consumption and CPU time consumption of the spdlog log library to print logs under different threads (threads) for multiple iterations. As the number of threads increases from 1 to 8, the spdlog log library does not show a stable downward trend, and even the time consumption increases when the number of threads increases in some cases, indicating that its multi-thread optimization is poor, and there may be problems such as large thread synchronization overhead and resource competition. The CPU time consumption increases significantly with the increase of the number of threads, from 1079 ns for single-thread to 6593 ns for eight threads, reflecting that the CPU resource consumption of the spdlog log library increases significantly in the multi-thread scenario, and there may be problems such as unreasonable resource scheduling or low code execution efficiency.

[0133] As can be seen, in the present embodiment, the producer and consumer threads cooperate to realize ultra-low latency asynchronous formatted log processing. The log library is written in pure C language, has simple structure, strong compatibility, and does not depend on high version compiler and new language standard. The performance is much higher than that of the spdlog log library, has excellent multi-thread scalability, can efficiently utilize multi-core CPU resources, and has stable CPU resource utilization, so that the log information can be quickly and stably persisted to a file, providing an efficient and reliable solution for program log recording.

[0134] Consistent with the above method embodiments, please refer to Figure 7 , Figure 7 is a structural schematic diagram of an electronic device 7 provided by an embodiment of the present application, as shown in the figure, the electronic device 7 comprises a processor 71, a communication interface 72, a memory 73, a communication bus 74, and one or more programs 731, wherein the one or more programs 731 are stored in the above-mentioned memory 73 and are configured to be executed by the above-mentioned processor 71, and the one or more programs 731 comprise instructions for executing any step in the above-mentioned method embodiments.

[0135] It should be noted that, for the above-mentioned method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other orders or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0136] The embodiments of the present application can divide the functional units of the electronic device according to the above-mentioned method examples, for example, each functional unit can be divided according to each function, or two or more functions can be integrated in one processing unit. The integrated unit can be realized in the form of hardware or software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical functional division. There can be another division method in actual implementation.

[0137] In addition, the embodiments of the present application also provide a computer storage medium which stores a computer program capable of being loaded and executed by a processor, such as the above-mentioned ultra-low delay asynchronous formatting log processing method. The computer readable storage medium includes, for example, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0138] It should be noted that, for the above-mentioned method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other orders or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0139] In several embodiments provided in the present application, it should be understood that the disclosed method, device and system can be implemented in other manners. For example, the described device embodiment is merely illustrative. For example, the division of the units is merely logical function division. In actual implementation, another division manner can be adopted. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0140] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be a physical unit independently, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of hardware plus software function unit.

[0141] The integrated unit in the form of software function unit can be stored in a computer readable storage medium. The software function unit is stored in a storage medium, and includes instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute part of steps of the method according to the embodiments of the present application. The storage medium includes a U disk, a mobile hard disk, a magnetic disk, an optical disk, a volatile memory or a non-volatile memory. The non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced SDRAM (ESDRAM), synch link DRAM (SLDRAM) and direct rambus RAM (DRRAM), etc.

[0142] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0143] The embodiments of the present application are described in detail above, and the principle and implementation mode of the present application are described by applying specific examples; the above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed; in conclusion, the content of the specification should not be understood as the limitation of the present application.

[0144] Although the present application has been disclosed with reference to the above examples, it is not intended to limit the present application. Any person skilled in the art, without departing from the spirit and scope of the present application, can easily think of changes or substitutions, and can make various changes and modifications, including combinations of different functions and implementation steps, including software and hardware implementations, all of which are within the scope of the present application.

Claims

1. An ultra-low latency asynchronous formatted log processing system, characterized in that: The system comprises: A first processing module is configured to call a log output function through a producer thread, the log output function including a format string and a parameter list, the format string being used to define a standard format for log information, the parameter list including initial log information generated in real time by the producer thread; and, based on a macro expansion operation, determine whether the initial log information satisfies the standard format; and, determine a data type corresponding to the initial log information, the data type including a non-string type and a string type; and, execute different storage strategies based on the data type to store the initial log information in a TLS lock-free buffer corresponding to the producer thread; The second processing module is configured to detect the initial log information from the TLS lock-free buffer through a consumer thread; asynchronously format the initial log information to obtain target log information; and write the target log information into a log file.

2. The ultra-low latency asynchronous formatted log processing system according to claim 1, characterized in that: The first processing module includes: a storage area determination unit, configured to determine the amount of the initial log information in the parameter list through a VA_NARG macro operation; and apply for a target cache area from the TLS lock-free buffer according to the number of parameters of the initial log information; A time recording unit, used to call the rdtsc instruction to record a timestamp; A type determination unit, configured to determine a data type corresponding to the initial log information; A storage unit, for executing a first storage strategy when the data type is a non-string type, so as to store the initial log information and the timestamp in the target cache area; and for executing a second storage strategy when the data type is a string type, so as to store the target address data and the timestamp corresponding to the initial log information in the target cache area.

3. The ultra-low latency asynchronous formatted log processing system according to claim 2, characterized in that: The first storage strategy is to calculate a first offset of the initial log information, and store the initial log information and the timestamp in the target cache area according to the first offset; The second storage strategy is to copy the initial log information to a specified area within the TLS lock-free buffer through an lmark macro operation; and, determine the target address data of the initial log information within the specified area; and, calculate a second offset of the target address data, and store the target address data and the timestamp in the target cache area according to the second offset.

4. The ultra-low latency asynchronous formatted log processing system according to claim 2 or 3, characterized in that: The first processing module further includes: An updating unit is configured to update a latest usage status of the TLS lock-free buffer, where the latest usage status is used to indicate whether the initial log information is newly stored in the target cache area.

5. The ultra-low latency asynchronous formatted log processing system according to claim 4, characterized in that: The second processing module includes: a detection unit, configured to determine whether the initial log information is newly stored in the target cache area according to the latest usage status of the TLS lock-free buffer; a prefix adding unit, configured to, upon detecting that the initial log information is newly stored in the target cache area, restore the microsecond time according to the timestamp recorded by the rdtsc instruction; and add a time prefix to the initial log information according to the microsecond time; a format processing unit, configured to format the initial log information according to the format string based on a vsnprintf function, so as to replace the placeholders according to the structure of the format string to generate the target log information, wherein the format string includes the placeholders; The log file unit is used to write the target log information into the log file.

6. The ultra-low latency asynchronous formatted log processing system according to claim 1, characterized in that: The system further comprises a format definition module, wherein: The format definition module is used to receive the standard format of the log information input by the user before calling the log output function through the producer thread, and to construct the format character string according to the standard format of the log information.

7. The ultra-low latency asynchronous formatted log processing system according to claim 6, characterized in that: The first processing module is further configured to output a warning message when it is determined based on the macro expansion operation that the initial log information does not meet the standard format, wherein the warning message is used to prompt the user to modify the standard format.

8. An ultra-low latency asynchronous formatted log processing method, characterized in that: Applied to the ultra-low latency asynchronous formatted log processing system according to any one of claims 1 to 7, the method comprises: Calling a log output function through a producer thread, the log output function including a format string and a parameter list, the format string being used to define a standard format of log information, the parameter list including initial log information generated in real time by the producer thread; and determining, based on a macro expansion operation, that the initial log information satisfies the standard format; Determine a data type corresponding to the initial log information, where the data type includes a non-string type and a string type; Executing different storage strategies according to the data type to store the initial log information in the TLS lock-free buffer corresponding to the producer thread; Detecting the initial log information from the TLS lock-free buffer by a consumer thread; Performing asynchronous formatting processing on the initial log information to obtain target log information; Write the target log information into a log file.

9. The method according to claim 8, characterized in that The executing different storage strategies according to the data type includes: Determining the number of the initial log information in the parameter list through a VA_NARG macro operation; and applying for a target cache area from the TLS lock-free buffer according to the number of parameters of the initial log information; Call the rdtsc instruction to record the timestamp; When the data type is a non-string type, a first storage strategy is executed to store the initial log information and the timestamp in the target cache area; and when the data type is a string type, a second storage strategy is executed to store the target address data and the timestamp corresponding to the initial log information in the target cache area.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: The computer program / instructions are executed by a processor to implement the steps of the method according to any one of claims 8 to 9.

Images