A message-driven asynchronous log processing method

Abstract

The application discloses a kind of asynchronous log processing methods based on message driving, including obtaining the log information of multiple user threads;First mutual exclusion is added to the user thread of multithreading concurrency;According to the log information of each user thread, select the memory block matched with the size of log information from static buffer, and add call lock to the selected memory block, to obtain the allocated memory block;Release call lock, and write corresponding log information into the allocated memory block after unlocking;Through call event notification call thread, so that call thread reads multiple log information from static buffer, and sends to multiple processing threads;Each processing thread controls the output format of log information by custom callback function.The application adopts double buffering mechanism, can realize multithreading reading and writing simultaneously, effectively avoid reading and writing thread mutual blocking, improve the efficiency of asynchronous log processing, guarantee the real-time performance and reliability of automatic driving system.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a message-driven asynchronous log processing method. Background Technology

[0002] As autonomous driving becomes the mainstream trend of future automobiles, various types of autonomous vehicles are constantly emerging. The core of an autonomous vehicle is the central computing platform, which is responsible for the vehicle's fusion perception, localization, planning and other computing tasks, and has extremely high real-time requirements for the autonomous driving system. However, in practical engineering applications, logs are inevitably needed to locate and analyze problems in the autonomous driving system and to track the real-time operating status of the autonomous driving system. In addition, there are frequent I / O operations during the log writing process, which will affect the system performance.

[0003] Existing logging systems include synchronous logging systems and asynchronous logging systems. Synchronous logging systems directly output print information to the console or log files, printing directly wherever information needs to be recorded. However, this direct output requires synchronous waiting for logs to be flushed to disk, causing business threads to be blocked and severely impacting system performance.

[0004] Asynchronous logging systems only support outputting message content in a specified format. When outputting strings, asynchronous logging systems directly perform copying and other related operations on the strings, making the process relatively simple. Furthermore, asynchronous logging systems adopt a simple multi-producer and single-consumer model, using a single buffer to provide read and write operations for both user threads and system threads.

[0005] However, asynchronous logging systems cannot support multiple output sources simultaneously. Neither the corresponding log level nor the log output format can be customized; they must rely on a fixed format, resulting in low log readability. Furthermore, string copying involves dynamic memory allocation and switching between user and kernel modes, leading to significant performance overhead and reduced efficiency. Additionally, since writing logs involves multiple threads reading and writing to the same buffer, the lack of a solution to the lock granularity issue results in increased lock granularity. Moreover, the inability to guarantee effective concurrent reading and writing leads to thread blocking and reduced log processing efficiency.

[0006] Log systems are crucial in fault location applications. While general log system designs do not require particularly high performance, autonomous driving systems, due to their real-time and reliability requirements, also have high concurrency and low latency application scenarios. In addition, log system design itself has a certain degree of complexity, making general log systems not very feasible for autonomous driving scenarios.

[0007] Therefore, a message-driven asynchronous log processing method is needed to effectively avoid mutual blocking between read and write threads, realize multi-threaded concurrent read and write of asynchronous logs, and improve the efficiency of asynchronous log processing. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides a message-driven asynchronous log processing method that effectively avoids mutual blocking between read and write threads, enables multi-threaded concurrent read and write of asynchronous logs, and improves the efficiency of asynchronous log processing. The technical solution is as follows:

[0009] This invention provides a message-driven asynchronous log processing method, comprising:

[0010] Retrieve log information from multiple user threads;

[0011] A first mutex lock is acquired for the user thread corresponding to the multi-threaded concurrent operation;

[0012] Based on the log information of each user thread, a memory block matching the size of the log information is selected from the static buffer, and a call lock is acquired on the selected memory block to obtain the allocated memory block; wherein, the static buffer includes multiple memory blocks, and the memory block is static memory pre-allocated in the static buffer;

[0013] Release the call lock and write the corresponding log information into the unlocked allocated memory block;

[0014] By invoking an event to notify the calling thread, the calling thread reads multiple log messages from the static buffer and sends them to multiple processing threads.

[0015] Based on the received log information, each processing thread controls the output format of the log information through a custom callback function; wherein, each processing thread is triggered by a corresponding processing event.

[0016] Furthermore, the log information is information to be output, including:

[0017] System-level messages and user-level content information.

[0018] Furthermore, after releasing the call lock and writing the corresponding log information to the unlocked allocated memory block, the method further includes:

[0019] Determine whether the allocated memory block is writable;

[0020] If the allocated memory block is writable, then continue writing the log information to the allocated memory block; wherein, the pointer to the memory block being written to points to the corresponding allocated memory block, and the allocated memory block is set to the memory block being written to;

[0021] If the allocated memory block is not writable, then the pointer to the memory block being written to is passed to the pointer to the memory block that is not writable, so that the pointer to the memory block that is not writable points to the allocated memory block, and the pointer to the memory block being written to is set to null; wherein, the allocated memory block is set to a memory block that is not writable.

[0022] Furthermore, after passing the pointer of the memory block being written to to the pointer of the memory block that is not writable to make the pointer of the memory block that is not writable point to the allocated memory block, and setting the pointer of the memory block being written to null, the method further includes:

[0023] A temporary memory block is introduced, and the pointer of the non-writable memory block is passed to the pointer of the temporary memory block, so that the pointer of the temporary memory block points to the allocated memory block; wherein, the allocated memory block is set as a temporary memory block;

[0024] Release the first mutex lock to allow the user thread to continue writing the log information;

[0025] The pointer to the memory block being written to, which was set to null, is reinitialized and points to the unused allocated memory block in the static buffer, so as to enable the user thread to continue writing.

[0026] Furthermore, before notifying the calling thread via an event to read multiple log messages from the static buffer and send them to multiple processing threads, the method further includes:

[0027] The temporary memory block is converted into linked list data for storage; wherein the head node of the temporary memory block is represented by a first readable memory block;

[0028] A second mutex lock is applied to the calling thread and the processing thread to prevent the first readable memory block from being modified.

[0029] The memory block containing the log information to be output is set as a locked memory block; wherein, the pointer of the first readable memory block is passed to the locked memory block.

[0030] Furthermore, the step of notifying the calling thread through an event call, causing the calling thread to read multiple log messages from the static buffer and send them to multiple processing threads, includes:

[0031] The invocation event is used to notify the invocation thread;

[0032] Set read locks on multiple locked memory blocks, and set the initial value of the read locks to the number of log messages to be read;

[0033] The calling thread reads multiple log messages from the corresponding locked memory block;

[0034] The log messages are sent one-to-one to the corresponding processing threads.

[0035] Furthermore, after the calling thread reads multiple log messages from the corresponding locked memory block, the method further includes:

[0036] After the read operation is complete, the locked memory block is released and reset to a free memory block in the static buffer for the next write operation.

[0037] Furthermore, after each processing thread controls the output format of the log information through a custom callback function based on the received log information, the method further includes:

[0038] Release the second mutex.

[0039] Furthermore, the log information is written and read through the TraceDictinary mechanism.

[0040] Furthermore, the TraceDictinary mechanism includes:

[0041] Each string in the log information is assigned an index number and an insertion number; wherein, the head pointer of the index number is the first index number, the head pointer of the insertion number is the first insertion number, and each index number corresponds to the first insertion number of the insertion number.

[0042] Implementing this invention has the following beneficial effects:

[0043] 1. This invention adopts a dual-buffering mechanism, which pre-allocates multiple memory blocks in the static buffer. While using some of the allocated memory blocks for multi-threaded simultaneous writing, it can also use other memory blocks in the static buffer for multi-threaded simultaneous reading. This effectively avoids mutual blocking between read and write threads, realizes multi-threaded concurrent reading and writing of asynchronous logs, greatly improves the efficiency of asynchronous log processing, and ensures the real-time performance and reliability of the autonomous driving system.

[0044] 2. This invention supports multiple output sources and allows each output source to be configured. By using custom callback functions, each output source can be customized, including the output format and output level of the log information, thereby improving the readability of the log information. Furthermore, it can completely decouple the output sources, resulting in high reliability, high scalability, and high flexibility.

[0045] 3. This invention adopts the TraceDictinary mechanism, which allows the log reading and writing process to operate only on the index number and insertion number of the string, avoiding direct string manipulation, reducing performance consumption, and further improving the efficiency of asynchronous log processing. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0047] Figure 1 A logical structure diagram of a message-driven asynchronous log processing method provided in an embodiment of the present invention;

[0048] Figure 2 A schematic diagram illustrating a method for determining the state of allocated memory blocks provided in an embodiment of the present invention;

[0049] Figure 3 A logical structure diagram of the method for implementing continued writing by a user thread provided in an embodiment of the present invention;

[0050] Figure 4 A logical structure diagram of a method for preventing the first readable memory block from being modified during the reading process, provided in an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram illustrating the process of reading log information from a locked memory block according to an embodiment of the present invention;

[0052] Figure 6 This is a logical structure diagram of the reuse of locked memory blocks provided in an embodiment of the present invention;

[0053] Figure 7 An architecture diagram of a message-driven logging system provided in an embodiment of the present invention;

[0054] Figure 8 A schematic diagram illustrating the principle of a static buffer provided in an embodiment of the present invention;

[0055] Figure 9 This is a schematic diagram illustrating the principle of the TraceDictinary mechanism provided in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments, and therefore should not be construed as limiting the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those shown in the figures or descriptions below. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0058] This embodiment addresses the problems in existing technologies, such as the inability of asynchronous logging systems to support multiple output sources simultaneously, resulting in low log readability; the inability to guarantee effective concurrent reading and writing when multiple threads operate on the same buffer; and the large memory consumption during string copying, leading to low overall efficiency of asynchronous logging systems. It provides a message-driven log processing method.

[0059] When multiple user threads need to write log information simultaneously, this message-driven log processing method acquires a first mutex lock on these concurrent user threads. This means the method exclusively accesses the corresponding buffers of these user threads (i.e., concurrent operations) for writing. Based on the log information of each user thread, a free memory block matching the size of the log information is selected from the static buffer. After acquiring a call lock on this free memory block, an allocated memory block is obtained. Then, the call lock is quickly released, allowing the corresponding log information to be written to this allocated memory block, until the allocated memory block becomes unwritable. The static buffer includes not only free allocated memory blocks but also unwritable memory blocks that have already been written with log information. To prevent log information in non-writable memory blocks from being modified during the reading process, a pointer is passed between a temporary memory block and the first readable memory block to the locked memory block. During the reading process, an event is invoked to notify the calling thread, enabling it to read multiple log messages from the locked memory block of the static buffer. Multiple processing events trigger corresponding processing threads, each controlling the output format of the corresponding log message through a custom callback function. This achieves multi-threaded output source output and customized output format. This process avoids mutual blocking between multi-threaded read and write threads, enabling effective concurrent multi-threaded read and write operations, significantly improving the efficiency of asynchronous log processing, and meeting the high real-time and reliability requirements of autonomous driving systems.

[0060] The technical solutions of the embodiments of the present invention will be described in detail below, with reference to the appendix to the specification. Figure 1 Instruction manual attached Figure 7 Included with instruction manual Figure 8 As shown, the method includes:

[0061] S101 retrieves log information from multiple user threads.

[0062] Specifically, log information consists of information to be output, including system-level messages (System information, hereinafter referred to as System info) and user-level content information (User information, hereinafter referred to as User info).

[0063] S103, acquire the first mutex lock on the user thread corresponding to the multi-threaded concurrent operation.

[0064] The acquisition of the first mutex lock here is an operation performed by user threads. User threads are multi-threaded concurrent operations, so this process involves acquiring the first mutex lock on the multi-threaded concurrent operation buffer.

[0065] In this embodiment, the first mutex lock is a HandlerMutex lock, which is used to lock the multi-threaded user threads so that when writing, the corresponding multi-threaded user thread can be selected for exclusive access, and the memory area corresponding to the user thread is marked to avoid subsequent access and write position errors.

[0066] S105, based on the log information of each user thread, select a memory block from the static buffer that matches the size of the log information, and add a call lock to the selected memory block to obtain the allocated memory block.

[0067] It should be noted that the static buffer itself is pre-allocated static memory. The static buffer includes various memory blocks with different functions and states, such as idle memory blocks, chunks being written to, finished writing chunks, temporary memory blocks, readable memory blocks, and locked memory blocks, etc.

[0068] For example, in this step, all selectable memory blocks are free memory blocks, meaning that the allocated memory blocks are still free memory blocks at this time, empty, and do not contain any written log information, and are in an idle state; in the subsequent S107 step, when the call lock is released, the allocated memory blocks are still free memory blocks, while the content in the allocated memory blocks where log information has been partially written is not empty, so the allocated memory blocks are in a writing state, and are memory blocks that are being written.

[0069] During the selection process, the size of the required free memory block can be determined based on the size of the log information. This allows the selection of a memory block that matches the log information size from among many free memory blocks. To distinguish this selected memory block and prevent subsequent write misalignment, a lock (LockMemory) is acquired on the selected free memory block, resulting in a locked allocated memory block and completing the memory block allocation. Subsequently, during write operations, the corresponding user thread can access the allocated memory block specifically and write the corresponding log information to it.

[0070] Furthermore, depending on the number of log messages to be written simultaneously, multiple free memory blocks can be allocated simultaneously, and call locks can be acquired on each of the multiple free memory blocks at the same time; and in the subsequent S107 step, the corresponding number of pre-allocated memory blocks can also be selected to perform the operation of writing log messages simultaneously based on the number of log messages (or the number of user threads), so as to realize multi-threaded simultaneous writing.

[0071] S107, release the call lock and write the corresponding log information to the unlocked allocated memory block.

[0072] Between steps S105 and S107, the acquisition and release of the call lock is a rapid process. The allocated memory block in the locked state only means that the memory block has been allocated. However, due to the existence of the call lock, the allocated memory block is in a non-writable state at this time. Therefore, after the call lock is released in this step (ReleaseMemory), the log information can be written to the allocated memory block.

[0073] It should be noted that multiple allocated memory blocks can be obtained simultaneously in step S105. Therefore, multiple allocated memory blocks can be written to simultaneously in step S105, so that multiple log messages are written to their respective allocated memory blocks, thus achieving multi-threaded simultaneous writing.

[0074] S109, by invoking an event to notify the calling thread, the calling thread reads multiple log messages from the static buffer and sends them to multiple processing threads.

[0075] Step S109 is the log information reading step, which mainly targets the memory block in the static buffer where log information has been written and is about to be output. Among the many memory blocks, the locked memory block is defined as a memory block that is no longer writable and locked so that it can no longer be modified. That is, the calling thread (dispatch thread) reads log information from the static buffer from the locked memory block. The locked memory block and the memory block being written to are different memory blocks in the static buffer. This allows the user thread to write log information to the memory block being written to and the calling thread to read other log information from the locked memory block at the same time. This dual-buffering mechanism, which uses different memory blocks for writing and reading, ensures the effective concurrency of reading and writing threads, avoids thread blocking, and greatly improves the efficiency of log processing.

[0076] In addition, when a memory block being written to reaches a state that is no longer writable after a period of writing, it will be converted into a locked memory block through a series of pointers, so that the calling thread can read log information from it and realize a round of writing and reading of the log information; that is, after the allocated memory block is no longer writable, a call event (DispatchEvent) will be sent to the calling thread to trigger the event, so that the calling thread can start reading the log information.

[0077] S111, based on the received log information, each processing thread controls the output format of the log information through a custom callback function; wherein, each processing thread is triggered by a corresponding processing event.

[0078] Existing asynchronous logging systems cannot support multiple output sources simultaneously. They cannot customize the log message levels or output formats, relying on fixed formats. In this new approach, after a calling thread reads a log message, a corresponding handler thread is triggered by a HandlerEvent. This handler thread processes the logs using callback functions, and each handler thread corresponds to a single output source (log message). By defining custom callback functions for different handler threads, customized control over the output formats of multiple log messages can be achieved. This allows for customization of the output format and level of each source through custom callback functions, completely decoupling the output sources and resulting in an asynchronous logging system with high reliability, scalability, and flexibility.

[0079] Specifically, in one possible implementation of this specification, as shown in the appendix... Figure 2 As shown, after releasing the call lock and writing the corresponding log information to the unlocked allocated memory block, i.e., after step S107, the method further includes:

[0080] S202, determine whether the allocated memory block is writable.

[0081] S204, if the allocated memory block is writable, then continue to write the log information to the allocated memory block; wherein, the pointer to the memory block being written points to the corresponding allocated memory block, and the allocated memory block is set as the memory block being written.

[0082] In this step, the pointer to the memory block being written points to the allocated memory block, indicating that the allocated memory block is writable or being written to, so that log information can be continuously written. After this step, for the memory block being written to, you can choose to always return to the above S202 step until the judgment result is that the memory block being written to is not writable, then execute the subsequent S206 step.

[0083] S206, if the allocated memory block is not writable, then the pointer of the memory block being written to is passed to the pointer of the memory block that is not writable, so that the pointer of the memory block that is not writable points to the allocated memory block, and the pointer of the memory block being written to is set to null; wherein, the allocated memory block is set to a memory block that is not writable.

[0084] Passing a pointer to a non-writable memory block indicates that the allocated memory block is currently in a non-writable state and can be set to a non-writable memory block. If the non-writable memory block is in a non-writable state, in order to prevent user threads from still operating on the non-writable memory block, the pointer that originally pointed to the memory block being written to is set to null. This means that there is currently no memory block being written to in the static buffer, and the non-writable memory block is no longer accessible or can not be written to.

[0085] Furthermore, in step S103, a Handlermutex lock is acquired for the entire concurrent user thread in this multi-threaded system. In one possible implementation of this specification, as shown in the appendix... Figure 8 As shown, both the non-writable memory block and the pointer to the memory block being written to are in a state of being locked by a Handlermutex. Although the pointer to the memory block being written to is null at this time, it is locked by the first mutex lock and is also in a state of being non-writable.

[0086] Specifically, after the aforementioned round of simultaneous write operations, memory blocks that are not writable under the Handlermutex lock and memory blocks that are being written to cannot be written, resulting in resource waste. Therefore, as follows... Figure 3 As shown, after step S206, where if the allocated memory block is not writable, the pointer to the memory block being written to is passed to the pointer to the non-writable memory block, so that the pointer to the non-writable memory block points to the allocated memory block, and the pointer to the memory block being written to is set to null, the method further includes:

[0087] S301, a temporary memory block is introduced, and the pointer of the non-writable memory block is passed to the pointer of the temporary memory block so that the pointer of the temporary memory block points to the allocated memory block; wherein the allocated memory block is set as a temporary memory block.

[0088] S303, release the first mutex lock so that the user thread can continue writing the log information.

[0089] S305, the pointer to the memory block being written to, which was set to null, is reinitialized and points to the unused allocated memory block in the static buffer, so as to enable the user thread to continue writing.

[0090] The temporary memory block is a memory block that temporarily stores log information. It exists for a very short time and is used to store log information that has been written by multiple user threads. As the number of user threads increases (or multiple rounds of user thread execution), more non-writable memory blocks are converted into temporary memory blocks through pointer passing from non-writable memory blocks to temporary memory blocks. That is, the log information in the temporary memory block is constantly updated in preparation for subsequent calling threads to read.

[0091] Introducing a temporary memory block allows for a single pointer pass, pointing to a static buffer. This prevents subsequent calling threads from operating on the non-writable memory block again after receiving a call event, thus reducing lock granularity.

[0092] After step S303, since the HandlerMutex lock is released, the user thread can continue writing to the log. At this time, the pointer to the memory block being written to is null, which means that if it is reinitialized, it can be reused to point to another unused free memory block in the static buffer. That is, the pointer to the memory block being written to can find and point to a new free memory block. This free memory block is the allocated memory block that was allocated after S101-S107 and the call lock was released. The pointer to the memory block being written to pointing to this free memory block means that the free memory block can be used for the next round of writing by the user thread.

[0093] That is, after passing the log information to the temporary memory block through the pointer, the pointers to the memory block being written to and the memory block that cannot be written to are released, so that the pointers to the memory block being written to and the memory block that cannot be written to can be applied to the next round of writing process of the user thread. Therefore, the message-driven asynchronous log processing method of the present invention can not only realize simultaneous writing by multiple threads, but also realize the reuse of memory blocks and realize multiple rounds of writing.

[0094] Furthermore, it should be noted that when reinitializing the pointer of the currently written memory block to find a new memory block for writing, it is necessary to determine whether there is a memory block in the static buffer that is in a writable state. If such a memory block exists (i.e., the memory block being written to), the write operation can be performed directly. If no memory block is being written to, it is necessary to further determine whether there is a free memory block in the current static buffer. If a free memory block exists, it can be allocated according to step S105 to obtain a new free allocated memory block. Then, the pointer of the reinitialized memory block being written to is pointed to this allocated memory block, so that the next round of log information writing operation can be performed in another memory block.

[0095] If at the current moment there are neither memory blocks being written to nor free memory blocks in the static buffer, then it is determined whether there are any non-writable memory blocks in the static buffer that can be reclaimed. If some memory blocks have no extra memory area for writing, or some memory blocks are too old, or some memory blocks can be refreshed, then these memory blocks can be determined to be reclaimable. For example, locked memory blocks that have completed all read operations in subsequent steps are reclaimable memory blocks. After clearing and releasing them, they can be used as free memory blocks for the next log information writing operation. If the memory blocks in the static buffer cannot be reclaimed, then a special memory block is selected to write the log information. However, in this case, the log information will never be output.

[0096] At this point, multiple threads are writing log information to the memory block being written to. Using a non-writable memory block as an intermediate medium, the pointer is passed to the temporary memory block, and the log information is ready to be read. However, since the contents of the temporary memory block need to be updated frequently, the memory block in a constantly updated state cannot be directly read, which may result in inaccurate reading.

[0097] Specifically, as shown in the instruction manual Figure 4 As shown, before the step of notifying the calling thread through an event to read multiple log messages from the static cache and send them to multiple processing threads, i.e., before step S109, the method further includes:

[0098] S402, the temporary memory block is converted into linked list data for storage; wherein, the head node of the temporary memory block is represented by a first readable memory block.

[0099] That is, after step S301, since the temporary memory block is frequently updated, it is converted into linked list data for storage. The updated log information is added to the linked list as linked list data. The head node of the temporary memory block is represented by the first readable memory block, and the linked list can be traversed through the head node to read all the log information in the temporary memory block. At this time, the log information in the first readable memory block (firstReadyChunk) will no longer change, and the first readable memory block is ready to be read.

[0100] S404, a second mutex lock is added to the calling thread and the processing thread to prevent the first readable memory block from being modified.

[0101] S406, the memory block containing the log information to be output is set as a locked memory block; wherein, the pointer of the first readable memory block is passed to the locked memory block.

[0102] To prevent the first readable memory block from being modified simultaneously by subsequent calling and processing threads, a second mutex is used to lock the calling and processing threads, protecting the first readable memory block and the locked memory block that needs to be output, until each calling thread has completed its read operation from the locked memory block. In this embodiment, the second mutex is a DispatchMutex, which can effectively protect the locked memory block, prevent the readable log information in the locked memory block from being modified during the reading process, and ensure the reliability of log information reading.

[0103] Specifically, after the DispatchMutex is locked, the calling thread can begin the read operation, as shown in the attached manual. Figure 5 As shown, the step of notifying the calling thread through an event, causing the calling thread to read multiple log messages from the static cache and send them to multiple processing threads, includes:

[0104] S501, the calling thread is notified through the calling event.

[0105] In fact, this step is before the temporary memory block is introduced in step S301 for pointer passing. After step S206 sets the pointer of the memory block being written to null, the calling thread has already received the call event being triggered. However, it cannot directly read from the non-writable memory block. So, after passing the pointer to the temporary memory block, the first readable memory block and the locked memory block in sequence, the subsequent step S503 is executed to read.

[0106] S503, set read locks on multiple locked memory blocks and set the initial value of the read locks to the number of log information to be read.

[0107] In this step, the initial value of the read lock is set to support multiple processing threads. Assuming there are n processing threads, the initial value of the read lock on the locked memory block is set to n, which means that the reading thread has read a total of n log messages. In the subsequent S505 step, n processing threads can process the n log messages one by one, realizing simultaneous reading by multiple threads.

[0108] S505, the calling thread reads multiple log messages from the corresponding locked memory block.

[0109] S507, the multiple log messages are sent one-to-one to the multiple processing threads.

[0110] Specifically, as shown in the instruction manual Figure 6 As shown, after the calling thread reads multiple log messages from the corresponding locked memory block, i.e., after step S505, the method further includes:

[0111] S602, the locked memory block after the read is completed is released and reset to the memory block in the static buffer for the next write.

[0112] In other words, the distinction between a locked memory block and a memory block being written to represents the state of that memory block. The states of different memory blocks can be transformed into each other, so that a memory block that has been used can be cleared and reused as a free memory block for the next round of writing. This is the aforementioned recycling and reuse of non-writable memory blocks. This recycling process makes the static buffer highly utilized and also helps to reduce the waiting process between different threads, thereby improving the overall processing efficiency.

[0113] Specifically, after each processing thread controls the output format of the received log information through a custom callback function, the method further includes:

[0114] Release the second mutex.

[0115] In this step, releasing DispatchMutex means releasing all DispatchMutexes, that is, releasing the DispatchMutexes on which the first readable memory block and the locked memory block depend. Furthermore, releasing DispatchMutexes must be performed after all memory blocks sent by the calling thread to the processing thread have completed their read operations, in order to ensure the stability of log information output and improve the efficiency of asynchronous log processing.

[0116] This invention employs a dual-buffering mechanism. Since the memory blocks used by the write and read operations (the memory block being written to and the locked memory block, respectively) are different memory blocks in the static buffer, simultaneous reading and writing will not cause the two threads to block each other, which can greatly improve the efficiency of log processing.

[0117] Specifically, this invention also introduces the TraceDictinary mechanism to implement the writing and reading of log information, thereby avoiding direct copying of strings, reducing performance consumption, and further improving the efficiency of asynchronous log processing.

[0118] As per the instruction manual Figure 9 As shown, most of the log information written consists of strings. Therefore, each string in the log information is assigned an index number (abbreviated as dictID) and an insertion number (abbreviated as entryID). When the written or read strings form a string linked list, the assigned multiple index numbers also form an index linked list. Correspondingly, multiple insertion numbers form an insertion linked list. Thus, the index number can be used to query the string, while the insertion number can be used to query the insertion position of the string in the entire linked list.

[0119] Here, the first index number (firstDictID) is a pointer to the head of the index list, so each index number in the index list can be found by using the first index number; similarly, the first insertion number (firstEntryID) is a pointer to the head of the insertion list, so each insertion number in the insertion list can be found by using the first insertion number.

[0120] When writing log information, the corresponding index number and insertion number are actually written to the index linked list and insertion linked list respectively. First, the first index number corresponding to the original index linked list is found, and the new index number is inserted in front of the first index number. Each index number corresponds to the first insertion number of the head pointer of the insertion linked list. The newly inserted insertion number is added to the head of the corresponding insertion linked list, completing the writing of the index number and insertion number. When it is necessary to read log information and perform string copying, only the corresponding index number and insertion number need to be operated on, without directly manipulating the string, which greatly saves performance. When it is necessary to query the actual content of the string corresponding to the dictionary, only the linked list needs to be traversed, which greatly improves the efficiency of log processing.

[0121] In one possible implementation of this specification, taking simultaneous writing and reading by three threads as an example, please refer to the appendix to the specification. Figure 7 If three user threads (User thread1, User thread2, and User thread3) need to write to a memory block simultaneously, the multi-threaded concurrent operation buffer is first locked with a HandlerMutex lock. Then, based on the System info and User info that each user thread needs to output, a memory block of matching size is allocated to each user thread from the static buffer. LockMemory is then called to lock the three memory blocks respectively, so that the corresponding user threads can write log information to the corresponding memory blocks after releasingMemory to unlock them.

[0122] Since the static buffer is pre-allocated static memory, please refer to the appendix of the manual. Figure 8This includes not only idleChunks and chunkBeingWritten, which can be used to write log information, but also lockedChunks and firstReadyChunk, which are used to read log information. In order to ensure that some memory blocks are written to without affecting the reading of other memory blocks, pointers are passed between different memory blocks. A double buffering mechanism is adopted. User threads can write log information to chunkBeingWritten, and use finishedWritingChunks as an intermediate medium to point the pointer to temporaryChunks. The temporaryChunks are converted into a linked list for storage, and firstReadyChunk represents the head node of the linked list for reading.

[0123] When a memory block reaches a write-in state, see [link / reference] Figure 7 The dispatch thread sends a DispatchEvent to trigger the event, causing the dispatch thread to read the log information written by User thread1, User thread2, and User thread3, and then sends it to the three processing threads (handler thread1, handler thread2, and handler thread3) through three processing events (HandlerEvent1, HandlerEvent2, and HandlerEvent3). In each processing thread, the output format of the log information is controlled by a custom callback function (Callback fun1, Callback fun2, and Callback fun3).

[0124] Furthermore, the entire write and read process employs the TraceDictionary mechanism; please refer to the attached manual. Figure 9 When User thread3 needs to write log information, it queries the head pointer firstDictID of the indexed linked list and inserts DictID3 into the head of DictID2. Then, according to the firstEntryID corresponding to DictID2, it inserts EntryID3 into the head of the insertion linked list. When reading the log information of User thread3, it only needs to operate on DictID3, which greatly improves the processing efficiency.

[0125] As can be seen from the above embodiments, the message-driven asynchronous log processing method in the embodiments of the present invention has the following beneficial effects:

[0126] 1. This invention adopts a dual-buffering mechanism, which pre-allocates multiple memory blocks in the static buffer. While using some of the allocated memory blocks for multi-threaded simultaneous writing, it can also use other memory blocks in the static buffer for multi-threaded simultaneous reading. This effectively avoids mutual blocking between read and write threads, realizes multi-threaded concurrent reading and writing of asynchronous logs, greatly improves the efficiency of asynchronous log processing, and ensures the real-time performance and reliability of the autonomous driving system.

[0127] 2. This invention supports multiple output sources and allows each output source to be configured. By using custom callback functions, each output source can be customized, including the output format and output level of the log information, thereby improving the readability of the log information. Furthermore, it can completely decouple the output sources, resulting in high reliability, high scalability, and high flexibility.

[0128] 3. This invention adopts the TraceDictinary mechanism, which allows the log reading and writing process to operate only on the index number and insertion number of the string, avoiding direct string manipulation, reducing performance consumption, and further improving the efficiency of asynchronous log processing.

[0129] This invention also provides a storage medium storing at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the message-driven asynchronous log processing method described above. Optionally, the storage medium may be located at at least one network server among multiple network servers in a computer network. Furthermore, the storage medium may include, but is not limited to, random access memory (RAM), read-only memory (ROM), USB flash drive, portable hard drive, disk storage device, flash memory device, other volatile solid-state storage devices, and other storage media capable of storing program code.

[0130] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0131] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0132] The above description is merely some embodiments of the present invention and is not intended to limit the present invention. Those skilled in the art should understand that the present invention can have various changes and improvements, and any modifications, equivalent substitutions and improvements made in accordance with the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A message-driven asynchronous log processing method based on a double buffer mechanism, characterized in that, The method comprises: acquiring log information of a plurality of user threads; applying a first mutual exclusion lock to the user threads corresponding to the plurality of thread concurrent operations; selecting, according to the log information of each of the user threads, a memory block matching the size of the log information from a static buffer, and applying a call lock to the selected memory block to obtain an allocated memory block; wherein the static buffer comprises a plurality of memory blocks, and the memory blocks are static memory pre-allocated in the static buffer; the static buffer comprises a writing memory block and a locked memory block; releasing the call lock and writing the corresponding log information into the allocated memory block after the call lock is released; the allocated memory block is the writing memory block; and the writing memory block is converted into the locked memory block; notifying a calling thread through a calling event, so that the calling thread reads a plurality of the log information from the locked memory block of the static buffer and sends the log information to a plurality of processing threads; controlling, by each of the processing threads, the output format of the log information through a self-defined callback function according to the received log information; wherein each of the processing threads is triggered by a corresponding processing event.

2. The message-driven asynchronous log processing method according to claim 1, wherein, The log information is to-be-output information, comprising: system-level messages and user-level content information.

3. The message-driven asynchronous log processing method according to claim 1, wherein, After the releasing of the call lock and the writing of the corresponding log information into the allocated memory block, the method further comprises: determining whether the allocated memory block is writable; if the allocated memory block is writable, continuing to write the log information into the allocated memory block; wherein a pointer of a writing memory block points to the corresponding allocated memory block, and the allocated memory block is set as the writing memory block; if the allocated memory block is not writable, transferring the pointer of the writing memory block to a pointer of a non-writable memory block, so that the pointer of the non-writable memory block points to the allocated memory block, and setting the pointer of the writing memory block as empty; wherein the allocated memory block is set as the non-writable memory block.

4. The message-driven asynchronous log processing method according to claim 3, wherein, After the transferring of the pointer of the writing memory block to the pointer of the non-writable memory block, so that the pointer of the non-writable memory block points to the allocated memory block, and setting the pointer of the writing memory block as empty, the method further comprises: introducing a temporary memory block, transferring the pointer of the non-writable memory block to a pointer of the temporary memory block, so that the pointer of the temporary memory block points to the allocated memory block; wherein the allocated memory block is set as the temporary memory block; releasing the first mutual exclusion lock, so that the user threads continue to write the log information; reinitializing the pointer of the writing memory block set as empty and pointing to the allocated memory block in the static buffer that is not used, for realizing the continuous writing of the user threads.

5. The message-driven asynchronous log processing method according to claim 4, wherein, Before the notifying of the calling thread through the calling event, so that the calling thread reads a plurality of the log information from the static buffer and sends the log information to the plurality of processing threads, the method further comprises: The temporary memory block is converted into a linked list data for storage, wherein the head node of the temporary memory block is represented by a first readable memory block; A second mutual exclusion lock is added to the calling thread and the processing thread, so that the first readable memory block is not modified; The memory block containing the log information to be output is set as a locked memory block, wherein the pointer of the first readable memory block is passed to the locked memory block.

6. The message-driven asynchronous log processing method according to claim 5, wherein, The calling thread is notified by the calling event, so that the calling thread reads a plurality of log information from the static buffer and sends them to a plurality of processing threads, which comprises: The calling thread is notified by the calling event; A read lock is set for a plurality of locked memory blocks, and the initial value of the read lock is set as the number of log information to be read; The calling thread reads a plurality of log information from the corresponding locked memory block; A plurality of log information is sent to a plurality of processing threads one by one.

7. The message-driven asynchronous log processing method according to claim 6, wherein, After the calling thread reads a plurality of log information from the corresponding locked memory block, the method further comprises: The locked memory block after reading is released and reset to the idle memory block in the static buffer for next time writing.

8. The message-driven asynchronous log processing method according to claim 7, wherein, After each processing thread controls the output format of the log information through a custom callback function according to the received log information, the method further comprises: The second mutual exclusion lock is released.

9. The message-driven asynchronous log processing method according to claim 1, wherein, The log information is written and read through the TraceDictinary mechanism.

10. The message-driven asynchronous log processing method according to claim 9, wherein, The TraceDictinary mechanism comprises: Each string in the log information is allocated an index number and an insertion number, wherein the head pointer of the index number is the first index number, the head pointer of the insertion number is the first insertion number, and each index number corresponds to the first insertion number of the insertion number.

Images