Log processing method and device, electronic equipment and storage medium
By using log key-value pair table and time difference judgment in log processing method, the problem of duplicate data in log files is solved, and efficient log storage and problem positioning is achieved.
Patent Information
- Application Number
- CN202310961229.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-01
- Publication Date
- 2025-02-14
AI Technical Summary
In production systems, there is a large amount of duplicate log information in the log file, which occupies a lot of disk space and increases the difficulty of problem location.
By using the log key value pair table in the log processing method, we obtain the pending log data and its time stamps, and judge whether the time difference is greater than the preset time period. If so, store the log data to the log file and update the log key value. Comparison table.
It effectively removes duplicate data in log files, avoids waste of disk space, reduces the difficulty of problem location, and ensures the continuity of log data.
Smart Images
Figure CN119440383A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a log processing method, device, electronic device and storage medium. Background Art
[0002] In a production system, log information is stored in a log file, so that in the event of a failure, the problem can be located based on the log information, thereby solving the technical problem. However, in a log file, the same log information usually appears repeatedly for many times, and redundant log information takes up a large amount of disk space and also increases the difficulty of problem location. In the related art, the log information can be hashed, and the log information can be deduplicated by comparing the hash value of the log information to obtain log information without duplicate data. However, the difficulty of problem location through the deduplicated log information is still high. Summary of the invention
[0003] In order to solve the above technical problems, the present application provides a log processing method, device, electronic device and storage medium.
[0004] According to a first aspect of the present application, a log processing method is provided, comprising:
[0005] Obtaining each to-be-processed log data and a timestamp corresponding to the to-be-processed log data from the log output queue in sequence;
[0006] If the log key-value pair table does not contain the log data to be processed, storing the log data to be processed in a log file; wherein the log key-value pair table includes the log data and a timestamp corresponding to the log data;
[0007] If the log key-value pair table contains the to-be-processed log data, determine the time difference between the timestamp corresponding to the to-be-processed log data in the log output queue and the timestamp corresponding to the log key-value pair table;
[0008] If the time difference is greater than a preset time period, the log data to be processed is stored in a log file;
[0009] The log data to be processed and the timestamp corresponding to the log data to be processed are updated to the log key-value pair table.
[0010] Optionally, before updating the to-be-processed log data and the timestamp corresponding to the to-be-processed log data to the log key-value pair table, the method further includes:
[0011] If the number of log key-value pairs included in the log key-value pair table reaches a preset number of key-value pairs, a target log key-value pair is selected from the log key-value pairs included in the log key-value pair table, and the target log key-value pair is deleted.
[0012] Optionally, selecting a target log key-value pair from the log key-value pairs included in the log key-value pair table includes:
[0013] The least recently used log key-value pair is selected from the log key-value pairs included in the log key-value pair table as the target log key-value pair by using the least recently used algorithm.
[0014] Optionally, before sequentially acquiring each to-be-processed log data and a timestamp corresponding to the to-be-processed log data from the log output queue, the method further includes:
[0015] The log output queue is cleared, and the cleared log output queue is used as the log write queue, and the log write queue containing log data is used as the log output queue.
[0016] Optionally, the log write queue includes log data output by the business thread;
[0017] The method further comprises:
[0018] Before the log output queue is cleared and the cleared log output queue is used as the log write queue, and the log write queue containing log data is used as the log output queue, a mutex lock and a condition variable are acquired, wherein the mutex lock is used to deny the service thread from accessing the log write queue during the process of clearing the log output queue and using the cleared log output queue as the log write queue and the log write queue containing log data as the log output queue; the condition variable is used to clear the log output queue and use the cleared log output queue as the log write queue and the log write queue containing log data as the log output queue when the log write queue is not empty;
[0019] After processing the log data to be processed in the log output queue, the mutex lock and the condition variable are released.
[0020] Optionally, the clearing operation on the log output queue and using the cleared log output queue as the log write queue, and using the log write queue containing log data as the log output queue, includes:
[0021] After receiving the log writing notification message sent by the business thread, the log output queue is cleared, and the cleared log output queue is used as the log writing queue, and the log writing queue containing log data is used as the log output queue.
[0022] Optionally, updating the to-be-processed log data and the timestamp corresponding to the to-be-processed log data to the log key-value pair table includes:
[0023] If the log key-value pair table does not contain the log data to be processed and the timestamp corresponding to the log data to be processed, add the log data to be processed and the timestamp corresponding to the log data to be processed to the log key-value pair table;
[0024] If the log key-value pair table already contains the log data to be processed and the timestamp corresponding to the log data to be processed, delete the log data to be processed and the timestamp corresponding to the log data to be processed already contained in the log key-value pair table, and add the log data to be processed and the timestamp corresponding to the log data to be processed to the log key-value pair table.
[0025] Optionally, the log data includes one or more of the following: log information, a line number of a code corresponding to the log information, a file name where the log information is located, and a function name where the log information is located.
[0026] According to a second aspect of the present application, a log processing device is provided, including:
[0027] A data acquisition module, used to sequentially acquire each to-be-processed log data and a timestamp corresponding to the to-be-processed log data from the log output queue;
[0028] A log output module, used for storing the log data to be processed in a log file if the log key-value pair table does not contain the log data to be processed; wherein the log key-value pair table includes the log data and the timestamp corresponding to the log data;
[0029] A time difference determination module, configured to determine a time difference between a timestamp corresponding to the to-be-processed log data in the log output queue and a timestamp corresponding to the to-be-processed log data in the log key-value pair table if the to-be-processed log data is included in the log key-value pair table;
[0030] The log output module is further used to store the to-be-processed log data into a log file if the time difference is greater than a preset time period;
[0031] The log key-value pair table updating module is used to update the to-be-processed log data and the timestamp corresponding to the to-be-processed log data to the log key-value pair table.
[0032] Optionally, the log processing device further includes:
[0033] A target log key-value pair selection module is used to select a target log key-value pair from the log key-value pairs contained in the log key-value pair table if the number of log key-value pairs contained in the log key-value pair table reaches a preset number of key-value pairs;
[0034] The target log key-value pair deletion module is used to delete the target log key-value pair.
[0035] Optionally, the target log key-value pair selection module is specifically configured to select the least recently used log key-value pair from the log key-value pairs included in the log key-value pair table as the target log key-value pair by using a least recently used algorithm.
[0036] Optionally, the log processing device further includes:
[0037] The log queue exchange module is used to clear the log output queue, and use the cleared log output queue as the log write queue, and use the log write queue containing log data as the log output queue.
[0038] Optionally, the log write queue includes log data output by the business thread;
[0039] The log processing device 400 further includes:
[0040] A mutual exclusion lock and condition variable acquisition module, used for acquiring a mutual exclusion lock and a condition variable before performing a clearing operation on the log output queue and using the cleared log output queue as a log write queue and using the log write queue containing log data as the log output queue, wherein the mutual exclusion lock is used to deny the service thread from accessing the log write queue during the process of performing a clearing operation on the log output queue and using the cleared log output queue as a log write queue and using the log write queue containing log data as the log output queue; the condition variable is used to perform a clearing operation on the log output queue and use the cleared log output queue as a log write queue and use the log write queue containing log data as the log output queue when the log write queue is not empty;
[0041] The mutex lock and condition variable release module is used to release the mutex lock and the condition variable after processing the log data to be processed in the log output queue.
[0042] Optionally, the log queue exchange module is specifically used to clear the log output queue after receiving the log write notification message sent by the business thread, and use the cleared log output queue as the log write queue, and use the log write queue containing log data as the log output queue.
[0043] Optionally, the log key-value pair table updating module is specifically used to add the pending log data and the timestamp corresponding to the pending log data to the log key-value pair table if the log key-value pair table does not contain the pending log data and the timestamp corresponding to the pending log data; if the log key-value pair table already contains the pending log data and the timestamp corresponding to the pending log data, delete the pending log data and the timestamp corresponding to the pending log data already contained in the log key-value pair table, and add the pending log data and the timestamp corresponding to the pending log data to the log key-value pair table.
[0044] Optionally, the log data includes one or more of the following: log information, a line number of a code corresponding to the log information, a file name where the log information is located, and a function name where the log information is located.
[0045] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, the processor being configured to execute a computer program stored in a memory, the computer program implementing the log processing method described in the first aspect when executed by the processor.
[0046] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the log processing method described in the first aspect is implemented.
[0047] According to a fifth aspect of the present application, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the log processing method described in the first aspect.
[0048] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:
[0049] In the embodiment of the present application, after obtaining each pending log data and the timestamp corresponding to the pending log data from the log output queue, the pending log data can be filtered by the log key-value pair table, that is, whether to store the pending log data in the log file is determined by the log key-value pair table, wherein the log data and the timestamp corresponding to the log data are included in the log key-value pair table. If the log key-value pair table does not contain the pending log data, it means that the log file does not contain the pending log data, and the pending log data can be directly stored in the log file. If the log key-value pair table contains the pending log data, it means that the log file has stored the pending log data, and the time difference between the timestamp corresponding to the pending log data in the log output queue and the timestamp corresponding to the log key-value pair table can be determined; if the time difference is greater than a preset time period (such as 1 second, etc.), the pending log data is stored in the log file; if the time difference is less than or equal to the preset time period, the pending log data is not stored in the log file. In this way, the same log data will not be stored in the preset time period, avoiding the difficulty of increasing the problem location due to too many repeated logs in the event of a failure. At the same time, for the same log data, the log data can be stored in different preset time periods to ensure the continuity of the log data and provide more references for problem location, thereby reducing the difficulty of problem location. In addition, by updating the log key-value pair table, the real-time nature of the timestamp corresponding to the log data to be processed in the log key-value pair table can be guaranteed. After obtaining the next log data to be processed, the same log data can be avoided from being repeatedly stored in the preset time period. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0052] Figure 1 A flow chart of the log processing method in the embodiment of the present application;
[0053] Figure 2 A schematic diagram of an exchange log queue in an embodiment of the present application;
[0054] Figure 3 This is another flow chart of the log processing method in the embodiment of the present application;
[0055] Figure 4A schematic diagram of the structure of a log processing device in an embodiment of the present application;
[0056] Figure 5 A schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0057] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0058] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.
[0059] See also Figure 1 , Figure 1 A flow chart of the log processing method in the embodiment of the present application may include the following steps:
[0060] Step S110, sequentially obtaining each to-be-processed log data and a timestamp corresponding to the to-be-processed log data from the log output queue.
[0061] In the embodiment of the present application, the log output queue may include multiple log data, for example, the log write queue includes the log data output by the service thread, and the service thread can output the timestamp of the log data when outputting the log data. Wherein, the service thread includes: network service thread, memory service thread, etc. In certain embodiments, API (Application Programming Interface, application program interface) function can be provided for other subsystems in the production system, and the service threads in other subsystems can output log data by calling the application program interface function, for example, the application program interface function is logInfo (msg), and when the service thread calls logInfo (msg), the log information msg can be added to the log input queue. The log input queue is the message queue used when the service thread outputs log data, and the log output queue is the message queue used when the log processing thread processes the log data output by the service thread, and the two can be the same message queue, or two different message queues. It should be noted that in order not to affect the normal operation of the service thread, a special thread (i.e. the aforementioned log processing thread) can be used in the embodiment of the present application to do log processing.
[0062] In some embodiments, if the log input queue and the log output queue are the same message queue, after the service thread writes log data in the log input queue, the log input queue can be processed by the log processing thread as the log output queue, and the log data in the log output queue can be written into the log file.
[0063] In some embodiments, if the log input queue and the log output queue are two different message queues, the log processing thread can clear the log output queue, and use the cleared log output queue as the log write queue, and use the log write queue containing log data as the log output queue, that is, by using a double buffer queue, and exchanging the double buffer queue to realize the processing of the log. Optionally, the service thread can send a log write notification message to the log processing thread, and after the log processing thread receives the log write notification message sent by the service thread, it clears the log output queue, and uses the cleared log output queue as the log write queue, and uses the log write queue containing log data as the log output queue.
[0064] For example, suppose the log input queue and log output queue are queue A and queue B respectively. In the initial state, both queue A and queue B are empty. The business thread can output log data to queue A. After the log processing thread detects that queue A is full, queue B is used as the log input queue and queue A filled with log data is used as the log output queue. The business thread can continue to output log data to queue B. The log processing thread can process the log data in queue A and clear queue A after processing. When the log processing thread detects that queue B is full, queue B is used as the log output queue and queue A is used as the log input queue after clearing it. This process repeats until all log data is processed.
[0065] See also Figure 2 , Figure 2 This is a schematic diagram of the exchange log queue in an embodiment of the present application. The business thread outputs log data to the upper queue. After the log processing thread detects that the queue is full, it obtains log data from the lower queue through the exchange queue.
[0066] According to the above method, while the business thread outputs log data to the log input queue, the log processing thread can process the log data in the log output queue and write the log data in the log output queue into a log file. In this way, the business thread and the log processing thread can be processed in parallel, saving the overhead of mutual exclusion / synchronization of read and write operations in a single buffer area, and avoiding the processing of each log data triggering the log processing thread. That is, the business thread that outputs the log does not transmit the log data to the back end separately, but instead transmits multiple log data into a queue and then transmits them to the back end, that is, batch processing of the log data process reduces the number of times the thread wakes up and reduces the overhead of switching between threads.
[0067] In some embodiments, before the log output queue is cleared, and the log output queue after clearing is used as the log write queue, and the log write queue containing log data is used as the log output queue, the log processing thread can obtain a mutex and a conditional variable, and the mutex is used to perform the log output queue clearing operation, and the log output queue after clearing is used as the log write queue, and the log write queue containing log data is used as the log output queue. The process, the service thread is refused to access the log write queue; the conditional variable is used to perform the log output queue clearing operation when the log write queue is not empty, and the log output queue after clearing is used as the log write queue, and the log write queue containing log data is used as the log output queue. In this way, by combining the use of the mutex and the conditional variable, it is possible to avoid that the log processing thread exchanges two queues when the service thread has not yet written the log data. In addition, it is possible to avoid that the log processing thread and the service thread access the log write queue at the same time, causing the log data in the log write queue to be abnormal. Accordingly, after the log processing thread has processed the pending log data in the log output queue, the log output queue is used as the log input queue again, and the mutex and the conditional variable can be released at this time. In this way, the business thread can continue to access the log input queue.
[0068] In the embodiment of the present application, after the emptied log output queue is used as the log write queue, the log processing thread can send a notification message of clearing the queue to the service thread, so that the service thread continues to output log data to the log write queue. In this way, by informing the service thread that the current log write queue is empty, the service thread is prevented from waiting blindly.
[0069] Step S120, determining whether the log key-value pair table contains log data to be processed.
[0070] In an embodiment of the present application, a log key-value pair table can be maintained, and the log key-value pair table includes log data and a timestamp corresponding to the log data. In the log key-value pair table, the key is the log data, and the value is the timestamp corresponding to the log data. In some embodiments, the log data may include one or more of the following: log information, the line number of the code corresponding to the log information, the file name where the log information is located, and the function name where the log information is located. In other words, when performing a deduplication operation, one or more of the log information, the line number of the code corresponding to the log information, the file name where the log information is located, and the function name where the log information is located can be directly deduplicated.
[0071] It should be noted that for each log data to be processed obtained by the log processing thread, the log data to be processed and the timestamp corresponding to the log data to be processed can be updated to the log key-value pair table. In this way, if the log key-value pair table does not contain the log data to be processed, it indicates that the log processing thread has not obtained the log data to be processed, and the log file does not contain the log data to be processed, and step S130 can be directly executed; if the log key-value pair table contains the log data to be processed, it indicates that the log processing thread has obtained the log data to be processed, and the log file also contains the log data to be processed, and step S140 can be executed to filter and remove duplicates of the log data to be processed through the log key-value pair table.
[0072] Step S130, storing the log data to be processed in a log file.
[0073] Step S140, determining the time difference between the timestamp corresponding to the to-be-processed log data in the log output queue and the timestamp corresponding to the to-be-processed log data in the log key-value pair table.
[0074] It is understandable that the log data to be processed is the latest log data obtained from the log output queue, and the timestamp corresponding to the log data to be processed in the log output queue is later than the timestamp corresponding to the log data to be processed in the log key-value pair table. By determining the time difference between the timestamp corresponding to the log data to be processed in the log output queue and the timestamp corresponding to the log key-value pair table, it is determined whether to store the log data to be processed in the log file.
[0075] Step S150, determining whether the time difference is greater than a preset time period.
[0076] If the time difference is greater than a preset time period (for example, 1 second), it means that the log file does not contain the log data to be processed and the timestamp corresponding to the log data to be processed within the preset time period, and step S130 can be executed; if the time difference is less than or equal to the preset time period, it means that the log file contains the log data to be processed and the timestamp corresponding to the log data to be processed within the preset time period, and the log data to be processed and the timestamp corresponding to the log data to be processed are no longer stored in the log file, and step S160 is executed.
[0077] Step S160: Update the log data to be processed and the timestamp corresponding to the log data to be processed into the log key-value pair table.
[0078] In an embodiment of the present application, each pending log data stored in the log key-value pair table can be unique. If the log key-value pair table does not contain the pending log data and the timestamp corresponding to the pending log data, the pending log data and the timestamp corresponding to the pending log data can be added to the log key-value pair table. If the log key-value pair table already contains the pending log data and the timestamp corresponding to the pending log data, the pending log data and the timestamp corresponding to the pending log data already contained in the log key-value pair table are deleted, and the pending log data and the timestamp corresponding to the pending log data are added to the log key-value pair table. In this way, it is ensured that the log key-value pair table does not contain duplicate log data, each log key-value pair is unique, and the timestamp corresponding to each log data is the latest.
[0079] In an optional implementation, each log data may be arranged in chronological order, that is, for each log data to be processed and the timestamp corresponding to the log data to be processed, the log data to be processed and the timestamp corresponding to the log data to be processed may be added at the end of the log key-value pair table.
[0080] The log processing method of the embodiment of the present application can filter the log data to be processed by the log key-value pair table, that is, determine whether to store the log data to be processed in the log file by the log key-value pair table, wherein the log data and the timestamp corresponding to the log data are included in the log key-value pair table. If the log key-value pair table does not contain the log data to be processed, it means that the log file does not contain the log data to be processed, and the log data to be processed can be directly stored in the log file. If the log key-value pair table contains the log data to be processed, it means that the log file has stored the log data to be processed, and the time difference between the timestamp corresponding to the timestamp in the log output queue and the timestamp corresponding to the log key-value pair table can be determined; if the time difference is greater than the preset time period (such as 1 second, etc.), the log data to be processed is stored in the log file; if the time difference is less than or equal to the preset time period, the log data to be processed is not stored in the log file. In this way, the same log data will not be stored in the preset time period, avoiding the difficulty of increasing the problem location due to too many repeated logs in the case of a failure. At the same time, for the same log data, the log data can be stored in different preset time periods to ensure the continuity of the log data and provide more references for problem location, thereby reducing the difficulty of problem location. In addition, by updating the log key-value pair table, the real-time nature of the timestamp corresponding to the log data to be processed in the log key-value pair table can be guaranteed. After obtaining the next log data to be processed, the same log data can be avoided from being repeatedly stored in the preset time period.
[0081] See also Figure 3 , Figure 3 A flow chart of the log processing method in the embodiment of the present application may include the following steps:
[0082] Step S310: Obtain each to-be-processed log data and a timestamp corresponding to the to-be-processed log data from the log output queue in sequence.
[0083] Step S320: determine that the log key-value pair table does not contain the log data to be processed.
[0084] Step S330: storing the log data to be processed in a log file.
[0085] Step S340, determining the time difference between the timestamp corresponding to the to-be-processed log data in the log output queue and the timestamp corresponding to the to-be-processed log data in the log key-value pair table.
[0086] Step S350, determining whether the time difference is greater than a preset time period.
[0087] Step S360: If the number of log key-value pairs included in the log key-value pair table reaches a preset number of key-value pairs, a target log key-value pair is selected from the log key-value pairs included in the log key-value pair table, and the target log key-value pair is deleted.
[0088] In an embodiment of the present application, the number of log key-value pairs that can be stored in the log key-value pair table is usually limited. When storing log key-value pairs, in addition to automatically deleting duplicate log key-value pairs, when the number of log key-value pairs reaches a preset number of key-value pairs, the target log key-value pairs can also be selected from the log key-value pair table and deleted.
[0089] In an optional implementation, the least recently used algorithm can be used to select the least recently used log key-value pair from the log key-value pairs contained in the log key-value pair table as the target log key-value pair, and delete the least recently used log key value. Among them, the least recently used algorithm eliminates data based on the historical access records of the data. Its core idea is that "if the data has been accessed recently, it is more likely to be accessed in the future." In the least recently used algorithm, the newly acquired data is inserted into the head of the linked list. Whenever the cache hits (i.e., the cached data is accessed), the data is moved to the head of the linked list. When the linked list is full, the data at the end of the linked list is discarded.
[0090] In an embodiment of the present application, the log key-value pair table can be used as a linked list, and the log key-value pairs contained therein can be arranged in chronological order, and the log data corresponding to the latest timestamp in the log key-value pair table is the head of the linked list, and the log data corresponding to the earliest timestamp is the tail of the linked list. When the log key-value pair table does not contain the log data to be processed, the log data to be processed and the timestamp corresponding to the log data to be processed can be inserted into the head of the linked list; when the log key-value pair table contains the log data to be processed, the log data to be processed and the timestamp corresponding to the log data to be processed in the log key-value pair table are deleted, and the log data to be processed and the timestamp corresponding to the log data to be processed obtained from the log output queue are inserted into the head of the linked list. When the number of log key-value pairs contained in the log key-value pair table reaches the preset number of key-value pairs, the log data corresponding to the earliest timestamp in the log key-value pair table and the earliest timestamp are deleted.
[0091] It should be noted that in addition to using the least recently used algorithm to select the target log key-value pair in the log key-value pair table, other methods can also be used to select the target log key-value pair. Optionally, the target log key-value pair can be selected from the log key-value pair table based on the attribute information of the log data to be processed. For example, different log data are divided according to their importance. The log key-value pair to which the log data with the lowest importance belongs also has the lowest importance. The log key-value pair with the lowest importance can be deleted as the target log key-value.
[0092] Step S370: Update the log data to be processed and the timestamp corresponding to the log data to be processed into the log key-value pair table.
[0093] After deleting a log key-value pair in step S360, a new log key-value pair may be inserted. It is understandable that after obtaining the next log data to be processed, a log key-value pair may be deleted first and then a new log key-value pair may be inserted.
[0094] The log processing method of the embodiment of the present application can use the least recently used algorithm to update the log key-value pairs in the log key-value pair table, thereby automatically deleting the log key-value pairs that are not frequently used in the log key-value pair table, and automatically maintaining the update of the data in the log key-value pair table. In this way, when filtering and deduplicating the log data to be processed through the log key-value pair table, a better deduplication effect can be achieved for the frequently used log data to be processed, avoiding the redundant printing of duplicate logs in a single log processing process, improving the validity of the log data, and avoiding the occurrence of a large number of duplicate logs submerging the key logs, thereby reducing the difficulty of problem locating.
[0095] Corresponding to the above method embodiment, the present application embodiment also provides a log processing device, see Figure 4 , the log processing device 400 includes:
[0096] The data acquisition module 410 is used to sequentially acquire each to-be-processed log data and a timestamp corresponding to the to-be-processed log data from the log output queue;
[0097] The log output module 420 is used to store the log data to be processed into a log file if the log key-value pair table does not contain the log data to be processed; wherein the log key-value pair table includes the log data and the timestamp corresponding to the log data;
[0098] The time difference determination module 430 is used to determine the time difference between the timestamp corresponding to the to-be-processed log data in the log output queue and the timestamp corresponding to the log key-value pair table if the log key-value pair table contains the to-be-processed log data;
[0099] The log output module 420 is further used to store the log data to be processed into a log file if the time difference is greater than a preset time period;
[0100] The log key-value pair table updating module 440 is used to update the log data to be processed and the timestamp corresponding to the log data to be processed into the log key-value pair table.
[0101] Optionally, the log processing device 400 further includes:
[0102] A target log key-value pair selection module is used to select a target log key-value pair from the log key-value pairs contained in the log key-value pair table if the number of log key-value pairs contained in the log key-value pair table reaches a preset number of key-value pairs;
[0103] The target log key-value pair deletion module is used to delete the target log key-value pair.
[0104] Optionally, the target log key-value pair selection module is specifically used to select the least recently used log key-value pair from the log key-value pairs included in the log key-value pair table as the target log key-value pair by using a least recently used algorithm.
[0105] Optionally, the log processing device 400 further includes:
[0106] The log queue exchange module is used to clear the log output queue, and use the cleared log output queue as the log write queue, and use the log write queue containing log data as the log output queue.
[0107] Optionally, the log write queue includes log data output by the business thread;
[0108] The log processing device 400 further includes:
[0109] The mutual exclusion lock and condition variable acquisition module is used to acquire the mutual exclusion lock and condition variable before clearing the log output queue and using the cleared log output queue as the log write queue and the log write queue containing log data as the log output queue. The mutual exclusion lock is used to deny the business thread from accessing the log write queue during the process of clearing the log output queue and using the cleared log output queue as the log write queue and the log write queue containing log data as the log output queue. The condition variable is used to clear the log output queue and use the cleared log output queue as the log write queue and the log write queue containing log data as the log output queue when the log write queue is not empty.
[0110] The mutex lock and condition variable release module is used to release the mutex lock and condition variable after processing the pending log data in the log output queue.
[0111] Optionally, the log queue exchange module is specifically used to clear the log output queue after receiving the log write notification message sent by the business thread, and use the cleared log output queue as the log write queue and the log write queue containing log data as the log output queue.
[0112] Optionally, a log key-value pair table update module is specifically used to add the pending log data and the timestamp corresponding to the pending log data to the log key-value pair table if the log key-value pair table does not contain the pending log data and the timestamp corresponding to the pending log data; if the log key-value pair table already contains the pending log data and the timestamp corresponding to the pending log data, delete the pending log data and the timestamp corresponding to the pending log data already contained in the log key-value pair table, and add the pending log data and the timestamp corresponding to the pending log data to the log key-value pair table.
[0113] Optionally, the log data includes one or more of the following: log information, a line number of the code corresponding to the log information, a file name where the log information is located, and a function name where the log information is located.
[0114] The specific details of each module or unit in the above device have been described in detail in the corresponding method, so they will not be repeated here.
[0115] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.
[0116] In an exemplary embodiment of the present application, an electronic device is also provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the above-mentioned log processing method in this exemplary embodiment.
[0117] Figure 5 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. It should be noted that: Figure 5 The electronic device 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0118] like Figure 5 As shown, electronic device 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In RAM 503, various programs and data required for system operation are also stored. Central processing unit 501, ROM 502 and RAM 503 are connected to each other via a bus 504. Input / output (I / O) interface 505 is also connected to bus 504.
[0119] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a local area network (LAN) card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.
[0120] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 509, and / or installed from a removable medium 511. When the computer program is executed by the central processing unit 501, various functions defined in the device of the present application are executed.
[0121] In an embodiment of the present application, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned log processing method is implemented.
[0122] It should be noted that the computer-readable storage medium shown in the present application may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency, etc., or any suitable combination of the above.
[0123] In an embodiment of the present application, a computer program product is also provided. When the computer program product is run on a computer, the computer executes the above-mentioned log processing method.
[0124] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0125] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A log processing method, characterized in that: The method comprises: Obtaining each to-be-processed log data and a timestamp corresponding to the to-be-processed log data from the log output queue in sequence; If the log key-value pair table does not contain the log data to be processed, storing the log data to be processed in a log file; wherein the log key-value pair table includes the log data and a timestamp corresponding to the log data; If the log key-value pair table contains the to-be-processed log data, determine the time difference between the timestamp corresponding to the to-be-processed log data in the log output queue and the timestamp corresponding to the log key-value pair table; If the time difference is greater than a preset time period, the log data to be processed is stored in a log file; The log data to be processed and the timestamp corresponding to the log data to be processed are updated to the log key-value pair table.
2. The method according to claim 1, characterized in that Before updating the to-be-processed log data and the timestamp corresponding to the to-be-processed log data to the log key-value pair table, the method further includes: If the number of log key-value pairs included in the log key-value pair table reaches a preset number of key-value pairs, a target log key-value pair is selected from the log key-value pairs included in the log key-value pair table, and the target log key-value pair is deleted.
3. The method according to claim 2, characterized in that The selecting a target log key-value pair from the log key-value pairs included in the log key-value pair table includes: The least recently used log key-value pair is selected from the log key-value pairs included in the log key-value pair table as the target log key-value pair by using the least recently used algorithm.
4. The method according to claim 1, characterized in that Before sequentially acquiring each to-be-processed log data and a timestamp corresponding to the to-be-processed log data from the log output queue, the method further includes: The log output queue is cleared, and the cleared log output queue is used as the log write queue, and the log write queue containing log data is used as the log output queue.
5. The method according to claim 4, characterized in that The log writing queue includes log data output by the service thread; The method further comprises: Before the log output queue is cleared and the cleared log output queue is used as the log write queue, and the log write queue containing log data is used as the log output queue, a mutex lock and a condition variable are acquired, wherein the mutex lock is used to deny the service thread from accessing the log write queue during the process of clearing the log output queue and using the cleared log output queue as the log write queue and the log write queue containing log data as the log output queue; the condition variable is used to clear the log output queue and use the cleared log output queue as the log write queue and the log write queue containing log data as the log output queue when the log write queue is not empty; After processing the log data to be processed in the log output queue, the mutex lock and the condition variable are released.
6. The method according to claim 5, characterized in that The method of clearing the log output queue and using the cleared log output queue as the log write queue and using the log write queue containing log data as the log output queue includes: After receiving the log writing notification message sent by the business thread, the log output queue is cleared, and the cleared log output queue is used as the log writing queue, and the log writing queue containing log data is used as the log output queue.
7. The method according to claim 1, characterized in that The updating of the to-be-processed log data and the timestamp corresponding to the to-be-processed log data to the log key-value pair table includes: If the log key-value pair table does not contain the log data to be processed and the timestamp corresponding to the log data to be processed, add the log data to be processed and the timestamp corresponding to the log data to be processed to the log key-value pair table; If the log key-value pair table already contains the log data to be processed and the timestamp corresponding to the log data to be processed, delete the log data to be processed and the timestamp corresponding to the log data to be processed already contained in the log key-value pair table, and add the log data to be processed and the timestamp corresponding to the log data to be processed to the log key-value pair table.
8. The method according to claim 1, characterized in that: The log data includes one or more of the following: log information, a line number of a code corresponding to the log information, a file name where the log information is located, and a function name where the log information is located.
9. A log processing device, characterized in that: The device comprises: A data acquisition module, used to sequentially acquire each to-be-processed log data and a timestamp corresponding to the to-be-processed log data from the log output queue; A log output module, used for storing the log data to be processed in a log file if the log key-value pair table does not contain the log data to be processed; wherein the log key-value pair table includes the log data and the timestamp corresponding to the log data; A time difference determination module, configured to determine a time difference between a timestamp corresponding to the to-be-processed log data in the log output queue and a timestamp corresponding to the to-be-processed log data in the log key-value pair table if the to-be-processed log data is included in the log key-value pair table; The log output module is further used to store the to-be-processed log data into a log file if the time difference is greater than a preset time period; The log key-value pair table updating module is used to update the to-be-processed log data and the timestamp corresponding to the to-be-processed log data to the log key-value pair table.
10. An electronic device, characterized in that: include: A processor, wherein the processor is used to execute a computer program stored in a memory, and when the computer program is executed by the processor, the log processing method according to any one of claims 1 to 8 is implemented.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the log processing method according to any one of claims 1 to 8 is implemented.