Multi-thread log lock-free output method and device and storage medium

Abstract

The invention discloses a multi-thread log lock-free output method and device and a storage medium, and the method comprises the steps that a hash table is created in a memory, and each slot position in the hash table comprises a plurality of nodes used for storing log output content; a plurality of service threads transmit log output contents generated by various services to a hash table, and ID values of the service threads and corresponding slot numbers in the hash table are obtained; storing the log output content transmitted by the service thread into a node under the slot number corresponding to the ID value of the service thread; and finally, regularly traversing log output contents in the nodes under all slot numbers, and transmitting the log output contents to a log file. According to the invention, in the process of searching the slot position, as the ID value of the service thread is unique, the selection of the slot position is unique, and the situation that a plurality of threads occupy one slot position is avoided, so that the conflict of the service threads is reduced, the concurrency efficiency of service logic is increased, and the concurrency and throughput of the service are improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a multi-threaded log lock-free output method, device and storage medium. Background Art

[0002] Multi-threaded business services are currently the mainstream application scenario. In the process of threaded business operation, the output of log content will naturally be applied. On the one hand, it is used to visualize the business process, and on the other hand, it is used to quickly locate abnormal situations when problems occur. Since the log file needs to be appended during the log output process, multiple threads operate on one file at the same time. The conventional operation is to lock the file resources while the log is output, thereby controlling the log output sequence. Therefore, the use of a lock mechanism to intersperse log output in the business logic process will have a significant impact on the operation of the business.

[0003] That is, when multiple business threads output to the same file, in order to control the basic recording order of the file, the writing process needs to be controlled in an orderly manner. This example describes that three threads have a record that needs to be recorded in the log file, and the file lock is used to control that only one thread can obtain the lock power at the same time and record the content to the log file. After completion, the lock is released, and then the remaining two threads randomly obtain the lock to complete the writing process, and so on, until all threads complete writing and stop.

[0004] It can be seen that although the lock-controlled writing process can functionally meet the order and completeness of the log output content, the sequential nature of the log brings obstacles to the concurrency of the business (no waiting between business threads), and this obstacle is irrelevant to the business itself, bringing unnecessary disaster to the throughput. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a multi-threaded log lock-free output method, device and storage medium, which solves the existing lock-controlled writing process. Although it can meet the order and completeness of the log output content from a functional point of view, it brings obstacles to the concurrency of the business. In the process of finding the slot, since the ID value of the business thread is unique, the selection of the slot is unique corresponding to it, and there will be no situation where multiple threads occupy one slot, thereby reducing the conflict of business threads, increasing the concurrency efficiency of business logic, and improving the concurrency and throughput of the business.

[0006] Technical solution: The present invention provides a multi-threaded log lock-free output method, comprising the following steps:

[0007] Create a hash table in memory, the hash table includes slots 0 to n-1, and each slot includes multiple nodes for storing log output content;

[0008] Multiple business threads transmit the log output content generated by various businesses to the hash table. The hash table obtains the ID value of each business thread and uses the ID value of each business thread to obtain the corresponding slot number in the hash table.

[0009] The hash table then stores the log output content transmitted by the business thread in chronological order in the node under the slot number corresponding to the business thread ID value; the log transmission thread regularly traverses the log output content in the nodes under all slot numbers according to the preset first time interval, and transmits it to the log file.

[0010] Furthermore, the method of obtaining the ID value of each service thread is to use the pthread_self function to obtain it.

[0011] Furthermore, the use of the ID value of each business thread to obtain the corresponding slot number in the hash table means: after obtaining the ID value of each business thread, the hash table performs a modulo operation on the ID value of each business thread with the total number of slots n to obtain the remainder m, and the remainder m is the corresponding slot number.

[0012] Furthermore, the method of using the ID value of each business thread to obtain the corresponding slot number in the hash table means: after obtaining the ID value of each business thread, the hash table compares whether the number of slots is less than the number of business threads. If the number of slots is less than the number of business threads, the difference between the number of slots and the number of business threads is calculated, and the hash table supplements the corresponding number of slots according to the difference. The hash table then matches the ID value of each business thread with the slot number one by one in sequence; otherwise, the hash table directly matches the ID value of each business thread with the slot number one by one in sequence.

[0013] Furthermore, the hash table also monitors in real time whether the nodes under all slot numbers have updated new log output content. When the nodes under a certain slot number have not updated new log output content within a preset second time interval, the hash table clears all data content in the node under the slot number and releases the correspondence between the slot number and the ID value of the business thread.

[0014] Furthermore, the preset first time interval is 1 to 5 minutes; the preset second time interval is 15 to 30 minutes.

[0015] The present invention also includes a computer device, which includes at least a processor and a memory, and the processor is used to implement the multi-threaded log lock-free output method as described in any one of the above claims when executing a computer program stored in the memory.

[0016] The present invention also includes a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the multi-threaded log lock-free output method as described in any one of the above claims.

[0017] Beneficial effects of the present invention:

[0018] The present invention creates a hash table in memory, the hash table obtains the ID value of each business thread, and uses the ID value of each business thread to obtain the corresponding slot number in the hash table; then the log output content transmitted by the business thread is stored in the node under the slot number corresponding to the business thread ID value in chronological order; finally, the log transmission thread traverses the log output content in the nodes under all slot numbers and transmits it to the log file. In the process of searching for the slot, since the ID value of the business thread is unique, the selection of the slot is unique corresponding to it, and there will be no situation where multiple threads occupy one slot, thereby reducing the conflict of the business thread, increasing the concurrent efficiency of the business logic, and improving the concurrency and throughput of the business. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of a method for writing with control in the prior art;

[0020] Figure 2 This is a schematic diagram of the multi-threaded log lock-free output method of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0022] like Figure 1 As shown, when multiple existing business threads are output to the same file, it is necessary to control the writing process with a lock, that is, to control only one thread to obtain the power of the lock at the same time by means of a file lock, and record the content to the log file. After completion, release the lock, and then the remaining threads randomly obtain the lock, and then complete the writing process, and so on, until all threads complete the writing of the content and stop. Although it can meet the order and completeness of the log output content functionally, because of the sequentiality of the log, it brings obstacles to the concurrency of the business (no waiting between business threads), and also hinders the throughput of the business. In view of this, the application proposes a multi-threaded log lock-free output method.

[0023] like Figure 2 As shown, the multi-threaded log lock-free output method of the present invention includes the following steps:

[0024] Create a hash table in the memory, the hash table includes slots 0 to n-1, and in this embodiment, slots 0 to 2 are used as an example, and each slot includes multiple nodes for storing log output content;

[0025] Multiple business threads transmit the log output content generated by various businesses to the hash table. The hash table obtains the ID value of each business thread. The ID value of each business thread is obtained by using the pthread_self function to ensure that the ID value of each business thread is unique.

[0026] The hash table then uses the ID value of each business thread to obtain the corresponding slot number in the hash table, which means: after the hash table obtains the ID value of each business thread, the hash table performs a modulo operation on the ID value of each business thread with the total number of slots n to obtain a remainder m, and the remainder m is the corresponding slot number; it can be seen that the slot number is calculated based on the ID value of each business thread. In the system process, the ID value of each business thread increases sequentially, so the remainder obtained by the modulo operation also increases sequentially. One thread occupies one slot. For example, the ID value of the first business thread is 8, and the total number of slots is 30. Then the modulo operation obtains a remainder 8, and 8 is the corresponding slot number; the ID value of the second business thread is 9, so the modulo operation obtains a remainder 9, and 9 is the corresponding slot number; and so on, one thread occupies one slot, which is of course particularly suitable for the situation where the number of slots is greater than the number of business threads;

[0027] Or it can also mean: after the hash table obtains the ID value of each business thread, the hash table compares whether the number of slots is less than the number of business threads. If the number of slots is less than the number of business threads, the difference between the number of slots and the number of business threads is calculated, and the hash table supplements the corresponding number of slots according to the difference. The hash table then matches the ID value of each business thread with the slot number one by one in sequence; otherwise, the hash table directly matches the ID value of each business thread with the slot number one by one in sequence. For example, the number of business threads is 20 and the total number of slots is 15. It is inevitable that two threads will occupy one slot. Then, the difference of 5 between the number of slots 15 and the number of business threads 20 is calculated first. The hash table supplements the corresponding number of slots according to the difference 5, so that the number of slots is the same as the number of business threads. The hash table then matches the ID value of each business thread with the slot number one by one in sequence, that is, the ID value of the first business thread corresponds to the slot number 1, and the ID value of the second business thread corresponds to the slot number 2; and so on, one thread occupies one slot; it can be seen that this process is the process of finding a unique slot. Since the ID value of the business thread is unique, the slot selection is uniquely corresponding to it, and there will be no situation where multiple threads occupy one slot.

[0028] The hash table of the present invention can also monitor in real time whether the nodes under all slot numbers have updated new log output content. When the nodes under a certain slot number have not updated new log output content within a preset second time interval, the preset second time interval can be 15 to 30 minutes, that is, the business thread corresponding to the slot may have stopped working, then the hash table clears all data content in the node under the slot number, and releases the corresponding relationship between the slot number and the ID value of the business thread, so that the slot number and the ID value of the subsequent business thread can be bound accordingly.

[0029] Then, the hash table stores the log output content transmitted by the business thread in the node under the slot number corresponding to the business thread ID value in chronological order. Of course, it can be stored in chronological order, priority order, etc., according to the actual situation. Finally, the log transmission thread regularly traverses the log output content in the nodes under all slot numbers according to the preset first time interval and transmits it to the log file. The preset first time interval is 1 to 5 minutes.

[0030] In summary, the present invention creates a hash table in memory, the hash table obtains the ID value of each business thread, and uses the ID value of each business thread to obtain the corresponding slot number in the hash table; then the log output content transmitted by the business thread is stored in the node under the slot number corresponding to the business thread ID value in chronological order; finally, the log transmission thread traverses the log output content in the nodes under all slot numbers and transmits it to the log file. In the present invention, since the ID value of the business thread is unique, the selection of the slot is unique corresponding to it, and there will be no situation where multiple threads occupy one slot, thereby reducing the conflict of business threads, increasing the concurrent efficiency of business logic, and improving the concurrency and throughput of business.

[0031] An embodiment of the present invention further provides a computer device, which includes at least a processor and a memory, wherein the memory stores a computer program, and the processor is used to implement the steps of the multi-threaded log lock-free output method of the embodiment of the present invention when executing the computer program stored in the memory.

[0032] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-threaded log lock-free output method of the embodiment of the present invention are implemented.

Claims

1. A multi-threaded log lock-free output method, characterized in that: The following steps are involved: Create a hash table in memory, the hash table includes slots 0 to n-1, and each slot includes multiple nodes for storing log output content; Multiple business threads transmit the log output content generated by various businesses to the hash table. The hash table obtains the ID value of each business thread and uses the ID value of each business thread to obtain the corresponding slot number in the hash table. The hash table then stores the log output content transmitted by the business thread in chronological order into the node under the slot number corresponding to the business thread ID value; The log transmission thread periodically traverses the log output content in the nodes under all slot numbers according to the preset first time interval, and transmits it to the log file.

2. According to claim 1, a multi-threaded log lock-free output method is characterized in that: The method of obtaining the ID value of each service thread is to use the pthread_self function to obtain it.

3. According to claim 1, a multi-threaded log lock-free output method is characterized in that: The method of using the ID value of each business thread to obtain the corresponding slot number in the hash table means that after obtaining the ID value of each business thread, the hash table performs a modulo operation on the ID value of each business thread with the total number of slots n to obtain a remainder m, and the remainder m is the corresponding slot number.

4. According to claim 1, a multi-threaded log lock-free output method is characterized in that: The method of using the ID value of each business thread to obtain the corresponding slot number in the hash table means: after obtaining the ID value of each business thread, the hash table compares whether the number of slots is less than the number of business threads. If the number of slots is less than the number of business threads, the difference between the number of slots and the number of business threads is calculated, and the hash table supplements the corresponding number of slots according to the difference. The hash table then sequentially matches the ID value of each business thread with the slot number one by one; otherwise, the hash table directly matches the ID value of each business thread with the slot number one by one in sequence.

5. According to claim 1, a multi-threaded log lock-free output method is characterized in that: The hash table also monitors in real time whether the nodes under all slot numbers have updated new log output content. When the nodes under a certain slot number have not updated new log output content within a preset second time interval, the hash table clears all data content in the node under the slot number and releases the corresponding relationship between the slot number and the ID value of the business thread.

6. A multi-threaded log lock-free output method according to claim 5, characterized in that: The preset first time interval is 1 to 5 minutes; the preset second time interval is 15 to 30 minutes.

7. A computer device, characterized in that: The computer device includes at least a processor and a memory, and the processor is used to implement the multi-threaded log lock-free output method as described in any one of claims 1 to 6 when executing a computer program stored in the memory.

8. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the multi-threaded log lock-free output method as described in any one of claims 1-6.

Images