Log data processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a log data processing method and device, electronic equipment and a storage medium, and is applied to the technical field of data processing.The method comprises the steps that first log data corresponding to a head node is obtained from a current log output queue, and a target log type of the first log data is determined; obtaining the total number of logs processed by the log processing process within a preset unit time length; obtaining the occurrence frequency of logs of the target log type in a preset unit time length; and when the total number of the logs is smaller than a preset number threshold and the number of times of occurrence of the logs is smaller than a preset number threshold, storing the first log data into a log file, and deleting the first log data from the log output queue.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a log data processing method, device, electronic device and storage medium. Background Art

[0002] In the production system, the log information corresponding to each business thread will be continuously stored in the log file, so that in the event of a failure, the problem can be located based on the log information, and the technical problem can be solved. However, when the failure continues to occur, the same log information will usually be repeated many times. The redundant log information will take up a lot of disk space and increase the difficulty of problem location. In addition, when the disk space is limited, the redundant log information may also flush out the truly effective logs. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides a log data processing method, device, electronic device and storage medium.

[0004] According to a first aspect of the present application, a log data processing method is provided, comprising:

[0005] Obtaining first log data corresponding to the head node from the current log output queue, and determining a target log type of the first log data;

[0006] Get the total number of logs processed by the log processing process within the preset unit time;

[0007] Obtain the number of occurrences of the target log type within the preset unit time;

[0008] When the total number of logs is less than a preset number threshold, and the number of occurrences of the logs is less than a preset number threshold, the first log data is stored in a log file, and the first log data is deleted from the log output queue.

[0009] Optionally, the method further includes: when the total number of logs is greater than or equal to a preset number threshold, or when the number of occurrences of the log is greater than or equal to a preset number threshold, deleting the first log data from the log output queue.

[0010] Optionally, before obtaining the total number of logs processed by the log processing process within a preset unit time, the method further includes: obtaining a timestamp of the first log data from the log output queue;

[0011] Obtain the timestamp corresponding to the previous log data of the first log data; the obtaining of the total number of logs processed by the log processing process within a preset unit time length includes: when the timestamp of the first log data and the timestamp corresponding to the previous log data are within a preset time period, controlling a log counter to add 1, the log counter being used to record the total number of logs, and the duration of the preset time period being the preset unit time length; when the timestamp of the first log data and the timestamp corresponding to the previous log data are not within the preset time period, clearing the log counter and then adding 1; and taking the count of the log counter as the total number of logs.

[0012] Optionally, the total number of logs is the number of logs processed by the log processing process and stored in the log file within the preset unit time.

[0013] Optionally, obtaining the number of log occurrences of the target log type within the preset unit time includes: when a target keyword exists in the log key-value pair table, taking the value corresponding to the target keyword as the number of log occurrences;

[0014] If the target keyword does not exist in the log key-value pair table, determining that the number of occurrences of the log is 0;

[0015] Among them, the log key-value pair table includes multiple groups of key-value pairs, each group of key-value pairs includes a keyword and a value corresponding to the keyword, the target keyword is used to represent the log information of the target log type, and the value corresponding to the target keyword is used to represent the number of occurrences of the log data of the target log type within a preset time period.

[0016] Optionally, the method further includes:

[0017] In the case where the target keyword exists in the log key-value pair table, after taking the value corresponding to the target keyword as the number of occurrences of the log, the method further comprises: in the case where the target keyword exists in the log key-value pair table, adding 1 to the value corresponding to the target keyword;

[0018] or,

[0019] When the target keyword does not exist in the log key-value pair table, after determining that the log occurrence count is 0, the method further includes: when the target keyword does not exist in the log key-value pair table, adding the target keyword to the log key-value pair table and adding 1 to the value corresponding to the target keyword.

[0020] Optionally, before obtaining the first log data corresponding to the head node from the current log output queue, the method further includes:

[0021] When all log data in the first log queue have been processed by the log processing thread and there is log data written by the business thread in the second log queue, the cleared first log queue will be used as the current log write queue, and the second log queue will be used as the current log output queue. The first log queue is the original log output queue, and the second log queue is the original log write queue.

[0022] Optionally, before obtaining the first log data corresponding to the head node from the current log output queue, the method also includes: when the log data in the first log queue have been processed by the log processing thread and there is log data written by the business thread in the second log queue, setting a mutex lock for the second log queue, and the mutex lock is used to deny the business thread access to the second log queue.

[0023] According to a second aspect of the present application, a log data processing device is provided, comprising:

[0024] The data acquisition module is used to obtain the first log data corresponding to the head node from the current log output queue, and determine the target log type of the first log data; obtain the total number of logs processed by the log processing process within a preset unit time; obtain the number of log occurrences of the target log type within the preset unit time;

[0025] The data processing module is used to store the first log data in a log file and delete the first log data from the log output queue when the total number of logs is less than a preset number threshold and the number of log occurrences is less than a preset number threshold.

[0026] Optionally, the data processing module is further used to: when the total number of logs is greater than or equal to a preset number threshold, or the number of log occurrences is greater than or equal to a preset number threshold, delete the first log data from the log output queue.

[0027] Optionally, the data acquisition module is further used to: before acquiring the total number of logs processed by the log processing process within a preset unit time, acquire the timestamp of the first log data from the log output queue; acquire the timestamp corresponding to the previous log data of the first log data;

[0028] The data acquisition module is specifically used to: obtain the total number of logs processed by the log processing process within a preset unit time length, including: when the timestamp of the first log data and the timestamp corresponding to the previous log data are within a preset time period, control the log counter to add 1, and the log counter is used to record the total number of logs, and the duration of the preset time period is the preset unit time length; when the timestamp of the first log data and the timestamp corresponding to the previous log data are not within the preset time period, clear the log counter and then add 1; and use the count of the log counter as the total number of logs.

[0029] Optionally, the total number of logs is the number of logs processed by the log processing process and stored in the log file within the preset unit time.

[0030] Optionally, the data acquisition module is specifically used to: the acquisition of the number of log occurrences of the target log type within the preset unit time includes: when the target keyword exists in the log key-value pair table, taking the value corresponding to the target keyword as the number of log occurrences; when the target keyword does not exist in the log key-value pair table, determining that the number of log occurrences is 0;

[0031] Among them, the log key-value pair table includes multiple groups of key-value pairs, each group of key-value pairs includes a keyword and a value corresponding to the keyword, the target keyword is used to represent the log information of the target log type, and the value corresponding to the target keyword is used to represent the number of occurrences of the log data of the target log type within a preset time period.

[0032] Optionally, the data processing module is also used to:

[0033] In the case where the target keyword exists in the log key-value pair table, after taking the value corresponding to the target keyword as the number of occurrences of the log, the method further comprises: in the case where the target keyword exists in the log key-value pair table, adding 1 to the value corresponding to the target keyword;

[0034] or,

[0035] When the target keyword does not exist in the log key-value pair table, after determining that the log occurrence count is 0, the method further includes: when the target keyword does not exist in the log key-value pair table, adding the target keyword to the log key-value pair table and adding 1 to the value corresponding to the target keyword.

[0036] Optionally, the device also includes: a log queue processing module; the log queue determination module is also used to, before the data acquisition module obtains the first log data corresponding to the head node from the current log output queue, when the log data in the first log queue has been processed by the log processing thread, and there is log data written by the business thread in the second log queue, use the cleared first log queue as the current log write queue, and use the second log queue as the current log output queue, the first log queue is the original log output queue, and the second log queue is the original log write queue.

[0037] Optionally, the log queue processing module is also used to set a mutex lock for the second log queue before obtaining the first log data corresponding to the head node from the current log output queue, if the log data in the first log queue has been processed by the log processing thread and there is log data written by the business thread in the second log queue, and the mutex lock is used to deny the business thread access to the second log queue.

[0038] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, the processor being configured to execute a computer program stored in a memory, the computer program implementing the log data processing method described in the first aspect when executed by the processor.

[0039] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the log data processing method described in the first aspect is implemented.

[0040] According to a fifth aspect of the present application, a computer program product is provided. When the computer program product is run on a computer, the computer is caused to execute the log data processing method described in the first aspect.

[0041] In an embodiment of the present application, the first log data corresponding to the head node is obtained from the current log output queue, and the target log type of the first log data is determined; the total number of logs processed by the log processing process within a preset unit time is obtained; the number of log occurrences of the target log type within the preset unit time is obtained; when the total number of logs is less than a preset number threshold, and the number of log occurrences is less than the preset number threshold, the first log data is stored in a log file, and the first log data is deleted from the log output queue. In this way, the total number of logs processed by the log processing process within the preset gear time is limited by the preset number threshold, and the number of log occurrences of the target log type (i.e., the same type as the first log data) within the preset unit time is limited by the preset number threshold, so that the number of logs processed by the log processing process within the preset unit time is small, and the first log data currently being processed will only be stored in the log file when the number of log data of the same type as the current first log data within the preset unit time is also small, thereby achieving log data flow limiting from two levels: the total number of logs and the number of log occurrences of the same type of logs within the preset unit time, avoiding a large number of redundant logs from occupying the disk space corresponding to the log file, and to a certain extent avoiding the effective logs from being flushed away. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A flow chart of a log data processing method provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of an exchange log queue provided in an embodiment of the present application;

[0046] Figure 3 A flowchart of another log data processing method provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of the structure of a log data processing device provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0050] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0051] Currently, when failures continue to occur, the same log information will usually be repeated multiple times. Redundant log information will take up a lot of disk space and increase the difficulty of problem location. In addition, when disk space is limited, redundant log information may flush out truly valid logs.

[0052] In order to solve the above problems, an embodiment of the present application provides a log data processing method, which implements log data flow limiting from two levels: the total number of logs and the number of log occurrences of the same type of logs within a preset unit time, thereby avoiding a large number of redundant logs occupying the disk space corresponding to the log files, and to a certain extent avoiding the effective logs from being flushed.

[0053] The log data processing method provided in the embodiment of the present application can be implemented by a log data processing device or an electronic device, and the log data processing device can be a functional module or functional entity in the electronic device for implementing the log data processing method. Exemplarily, the electronic device can be a server, a computer, etc.

[0054] Figure 1 A flowchart of a log data processing method provided in an embodiment of the present application.

[0055] See also Figure 1 The log data processing method may include but is not limited to the following steps:

[0056] 101. Obtain the first log data corresponding to the head node from the current log output queue.

[0057] In the embodiment of the present application, the log output queue is a message queue used when the log processing thread processes the log data output by the business thread. The log output queue may include multiple log data, which may be log data output by different business threads. When the business thread outputs the log data, it may also output the timestamp of the log data. Among them, the business threads include: network business threads, memory business threads, etc.

[0058] In certain embodiments, API (Application Programming Interface, application program interface) function can be provided for other subsystems in production system, the service thread in other subsystem can output log data by calling application program interface function, for example, application program interface function is logInfo (msg), when service thread calls logInfo (msg), log information msg can be added to log input queue. The message queue used when log input queue is the service thread output log data, the message queue used when log output queue is the log processing thread to process the log data of service thread output, both can be the same message queue, also can be two different message queues.It should be noted that, in order not to affect the normal operation of service thread, in the embodiment of the present application, special thread (i.e. aforesaid log processing thread) can be used to do log processing.

[0059] In some embodiments, if the log input queue and the log output queue are the same message queue, after the service thread writes log data in the log input queue, the log input queue can be processed by the log processing thread as the log output queue, and the log data in the log output queue can be written into the log file.

[0060] In some embodiments, if the log input queue and the log output queue are two different message queues, the log processing thread can detect the amount of log data in the log write queue; if the amount of log data in the log write queue reaches a preset number of logs, the cleared log output queue can be used as the log write queue, and the log write queue full of log data can be used as the log output queue, that is, the log processing process is realized by using double buffer queues and exchanging the double buffer queues.

[0061] Figure 2 A schematic diagram of an exchange log queue provided in an embodiment of the present application.

[0062] For example, assume that there are two log queues, such as Figure 2 In the queue A and queue B shown, in the initial state, both queue A and queue B are empty. The business thread can write log data to queue A. After the log processing thread detects that queue A is full, queue B is used as the log input queue, and queue A filled with log data is used as the log output queue. The business thread can continue to output log data to queue B. The log processing thread can process the log data in queue A and clear queue A after processing. When the log processing thread detects that queue B is full, queue B is used as the log output queue and the cleared queue A is used as the log input queue, and so on.

[0063] In certain embodiments, before the log output queue after being emptied is used as the log write queue, and the log write queue full of log data is used as the log output queue, the log processing thread can obtain a mutex lock, and the mutex lock is used for the log processing thread to refuse the service thread to access the log write queue when the log output queue after being emptied is used as the log write queue, and the log write queue full of log data is used as the process of the log output queue. That is, it is possible to avoid the log processing thread and the service thread from accessing the log write queue at the same time, causing the log data in the log write queue to be abnormal. Accordingly, after the log processing thread has processed the pending log data in the log output queue, the log output thread is used as the log input thread again, and the mutex lock can be released at this moment. In this way, the service thread can continue to access the log input thread. In the embodiment of the present application, after the log output queue after being emptied is used as the log write queue, the log processing thread can send the notification information of the emptied queue to the service thread, so that the service thread continues to output log data to the log write queue. In this way, by informing the service thread that the current log write queue is empty, it is possible to avoid the blind waiting of the service thread.

[0064] In other embodiments, if the log input queue and the log output queue are two different message queues, and the log data in the first log queue has been processed by the log processing thread, and there is log data written by the business thread in the second log queue, the cleared first log queue will be used as the current log write queue, and the second log queue will be used as the current log output queue, the first log queue is the original log output queue, and the second log queue is the original log write queue.

[0065] For example, assume that there are two log queues, such as Figure 2 As shown in the queue A and queue B, in the initial state, queue A and queue B are both empty, and the business thread can write log data to queue A (that is, queue B is first used as the log input queue). After the log processing thread detects that queue B is empty and there is log data written by the business thread in queue A, the log queues can be exchanged, and queue A is used as the log output queue and queue B is used as the log input queue. That is, the business thread writes log data to queue B, and the log processing thread processes the log data in queue A. After the log processing thread completes the processing of the log data in queue A, queue A is cleared, and after detecting that there is log data written by the business thread in queue B, the log queues are exchanged again, and queue B is used as the log output queue and queue A is used as the log input queue, and so on.

[0066] In certain embodiments, the daily log data in the first log queue has all been processed by the log processing thread, and in the second log queue, there is the daily log data that the service thread writes, the first log queue after being emptied is used as the current log write queue, and on the basis of using the second log queue as the current log output queue, the mutex lock for the second log queue can also be set, and the mutex lock is used to refuse the service thread to access the second log queue. In other words, the log processing thread and the service thread can be avoided from accessing the log write queue simultaneously, causing the daily log data in the log write queue to be abnormal. Accordingly, the log processing thread, after processing the pending log data in the log output queue, the log output thread is used as the log input thread again, and can release the mutex lock at this moment. Like this, the service thread can continue to access the log input thread.

[0067] In the embodiment of the present application, after the emptied log output queue is used as the log write queue, the log processing thread can send a notification message of clearing the queue to the service thread, so that the service thread continues to output log data to the log write queue. In this way, by informing the service thread that the current log write queue is empty, the service thread is prevented from waiting blindly.

[0068] According to the above-described embodiment, by arranging two log queues and adopting the mode of exchanging log queues, when the business thread outputs the log data to the log input queue, the log processing thread can process the log data in the log output queue, and writes the log data in the log output queue to the log file.Like this, the business thread and the log processing thread can be processed in parallel, and the overhead of read and write operation mutual exclusion / synchronization when saving the single buffer area has been avoided, and the processing of each log data all triggers the log processing thread.That is, the business thread of the output log is not to send the log data to the rear end respectively, but to send the rear end after the multiple log data are pieced into the queue, that is, to the log data process batch processing, has reduced the number of times of thread awakening, and has reduced the overhead of switching between threads.

[0069] 102. Determine a target log type of first log data.

[0070] In an embodiment of the present application, the target log type corresponding to the first log data can be determined by obtaining log information from the first log data. The log information includes one or more of the following information corresponding to the log data: function name, line number of the code printing the log, file name, log string, part of the log string.

[0071] Different types of log data can be distinguished by different log information.

[0072] 103. Get the total number of logs processed by the log processing process within a preset unit time.

[0073] In some embodiments, the total number of logs processed by the log processing process may be the sum of a first number and a second number, wherein the first number is the number of log data processed by the log processing process and stored in the log file, and the second number is the number of log data processed by the log processing process and not stored in the log file.

[0074] In some embodiments, the total number of logs processed by the log processing process may be the first number, that is, the number of log data processed by the log processing process and stored in the log file within the preset unit time.

[0075] In some embodiments, before obtaining the total number of logs processed by the log processing process within a preset unit time, the timestamp of the first log data can also be obtained from the log output queue; the timestamp corresponding to the previous log data of the first log data is obtained; and then when the total number of logs processed by the log processing process within the preset unit time is obtained, the total log data can be recorded based on the log counter.

[0076] The method of recording the total log data based on the log counter may include:

[0077] If the timestamp of the first log data and the timestamp corresponding to the previous log data are within the preset time period, the log counter is controlled to increase by 1. If the timestamp of the first log data and the timestamp corresponding to the previous log data are not within the preset time period, the log counter is cleared and then increased by 1; the count of the above log counter is used as the total number of logs.

[0078] The log counter is used to record the total number of logs, and the duration of the preset time period is the preset unit duration. The log counter can be reset after each preset unit duration, and re-record the total number of logs corresponding to the next preset unit duration.

[0079] The previous log data is the most recent log data processed by the log processing process before the first log data.

[0080] The preset time period may be a specified time range, and the duration corresponding to the time range is a preset unit duration, for example, between 10:30:30 and 10:30:31, and the preset unit duration is 1 second.

[0081] In the embodiment of the present application, the log counter can record the total number of logs accumulated per second according to the preset unit time length of 1 second, and clear the data recorded in the previous second after entering the next second. For example, between 10:30:30 and 10:30:31, the total number of logs recorded can be 20. When entering the next second, that is, the time period between 10:30:31 and 10:30:32, the 20 recorded in the log counter can be cleared, and the total number of logs corresponding to the time period can be counted again.

[0082] 104. Obtain the number of occurrences of logs of the target log type within a preset unit time.

[0083] In some embodiments, obtaining the number of log occurrences of a target log type within a preset unit time may include but is not limited to: when a target keyword exists in the log key-value pair table, using the value corresponding to the target keyword as the number of log occurrences; when the target keyword does not exist in the log key-value pair table, determining the number of log occurrences to be 0; wherein the target keyword is used to represent the log information of the target log type, and the value corresponding to the target keyword is used to represent the number of occurrences of log data of the target log type within a preset time period.

[0084] The log key-value pair table may be a hash table, which may include multiple key-value pairs, each of which includes a keyword and a value corresponding to the keyword. In the log key-value pair table, the keyword of each key-value pair is used to represent log information of a log type, and the value corresponding to the keyword is the number of occurrences of log data of this log type within a preset unit time length.

[0085] The log information includes one or more of the following information corresponding to the log data: function name, line number of the code that prints the log, file name, log string, part of the log string. In other words, when classifying the log data, the classification can be based on one or more combinations of the function name, line number of the code that prints the log, file name, log string, part of the log string.

[0086] In some embodiments, when the target keyword exists in the log key-value pair table, after taking the value corresponding to the target keyword as the number of occurrences of the log, the value corresponding to the target keyword can be increased by 1 when the target keyword exists in the log key-value pair table. The existence of the target keyword in the log key-value pair table indicates that log data of the same log type as the first log data has appeared within the preset unit time length, so after the first log data of the same log type appears again, the value corresponding to the target keyword can be increased by 1, that is, the number of occurrences of the log data of the target log type in the preset unit time length is accumulated.

[0087] In some embodiments, when the target keyword does not exist in the log key-value pair table, after determining that the number of log occurrences is 0, if the target keyword does not exist in the log key-value pair table, a new target keyword can be added to the log key-value pair table, and the value corresponding to the target keyword can be increased by 1. Among them, the absence of the target keyword in the log key-value pair table indicates that log data of the same log type as the first log data has not appeared within the preset unit time length, so after the first log data of the same log type appears for the first time, the target keyword can be added to the log key-value pair table, and the value corresponding to the target keyword can be increased by 1, that is, the number of occurrences of log data of the target log type in the preset unit time length can be counted.

[0088] The above-mentioned preset unit time length can be 1 second, 2 seconds, 3 seconds, etc. The specific time length value of the preset unit time length is not limited in the embodiment of the present application and can be set according to actual needs.

[0089] In some embodiments, the target log type may refer to any type of log. The target log type may also refer to a log type that requires long-term sampling and monitoring.

[0090] When the above-mentioned target log type may also refer to a log type that requires long-term sampling and monitoring, only the key-value pairs of the log type that requires long-term sampling and monitoring may be recorded in the above-mentioned log key-value pair table, and the number of occurrences of log data for this log type is limited to a preset unit time length.

[0091] 105. Determine whether the total number of logs is less than a preset number threshold.

[0092] The preset number threshold may be a maximum allowed value of the number of logs processed by the log processing process within a preset unit time.

[0093] In some embodiments, the preset number threshold may be an empirical value set based on experience, or a maximum number of logs processed by the log processing process within a preset unit time obtained by statistics in a scenario where no failures occurred in history.

[0094] When it is determined that the total number of logs is less than the preset number threshold, the following steps 106 to 108 may be executed; when it is determined that the total number of logs is greater than or equal to the preset number threshold, the following step 108 may be directly executed.

[0095] 106. Determine whether the log occurrence count is less than a preset count threshold.

[0096] The preset number threshold may be the maximum allowed number of occurrences of the target log type within a preset unit time.

[0097] In some embodiments, the preset number threshold may be an empirical value set based on experience, or a maximum number of occurrences of the target log type within a preset unit time obtained by statistics in a scenario where no failures have occurred in history.

[0098] When it is determined that the log occurrence count is less than the preset count threshold, the following step 107 may be executed; when it is determined that the log occurrence count is greater than or equal to the preset count threshold, the following step 108 may be directly executed.

[0099] It should be noted that the order of step 105 and step 106 is not limited in the embodiment of the present application. That is to say, in the embodiment of the present application, the above step 105 may be performed first and then the above step 106, or the above step 106 may be performed first and then the above step 105. Figure 1 In the description, the example of executing step 105 first and then executing step 106 is used.

[0100] 107. Store the first log data in a log file, and delete the first log data from the log output queue.

[0101] 108. Delete the first log data from the log output queue, and do not store the first log data in the log file.

[0102] The log data processing method provided by the embodiment of the present application obtains the first log data corresponding to the head node from the current log output queue, and determines the target log type of the first log data; obtains the total number of logs processed by the log processing process within a preset unit time; obtains the number of log occurrences of the target log type within the preset unit time; when the total number of logs is less than a preset number threshold, and the number of log occurrences is less than the preset number threshold, stores the first log data in a log file, and deletes the first log data from the log output queue. In this way, the total number of logs processed by the log processing process within the preset gear time is limited by the preset quantity threshold, and the number of log occurrences of the target log type (i.e., the same type as the first log data) within the preset unit time is limited by the preset number threshold, so that the number of logs processed by the log processing process within the preset unit time is small, and the first log data currently being processed will only be stored in the log file when the number of log data of the same type as the current first log data within the preset unit time is also small, thereby achieving log data flow limiting from two levels: the total number of logs and the number of log occurrences of the same type of logs within the preset unit time, avoiding a large number of redundant logs from occupying the disk space corresponding to the log file, and to a certain extent avoiding the effective logs from being flushed away.

[0103] In the embodiment of the present application, correspondingly, when the total number of logs is greater than or equal to the preset number threshold, or the number of log occurrences is greater than or equal to the preset number threshold, the first log data is deleted from the log output queue, and the first log data is not stored in the log file. In this way, the total number of logs processed by the log processing process within the preset gear duration is limited by the preset number threshold, and the number of log occurrences of the target log type (i.e., the same type as the first log data) within the preset unit duration is limited by the preset number threshold, so that when the number of logs processed by the log processing process within the preset unit duration is too large, or the number of log data of the same type as the current first log data within the preset unit duration is too large, the first log data currently being processed in the log output queue can be deleted, thereby achieving the flow limitation of log data from the two levels of the total number of logs and the number of log occurrences of the same type of logs within the preset unit time, avoiding a large number of redundant logs from occupying the disk space corresponding to the log file, and avoiding the effective logs from being flushed to a certain extent.

[0104] Figure 3 A flowchart of another log data processing method provided in an embodiment of the present application.

[0105] like Figure 3 As shown, the log data processing method may include but is not limited to the following steps:

[0106] 301. Obtain first log data corresponding to the head node of the log output queue.

[0107] In an embodiment of the present application, when processing data in a log output queue, each log data will be dequeued from the log output queue after processing, that is, the log data will be deleted from the log output queue. Therefore, in an embodiment of the present application, the log data corresponding to the head node of the log output queue can be processed each time.

[0108] 302. Obtain a timestamp of the first log data.

[0109] 303. Compare the timestamp of the first log data with the timestamp corresponding to the previous log data to see whether they belong to the same second.

[0110] For example, assuming that the timestamp of the first log data and the timestamp corresponding to the previous log data are between 10:30:30 and 10:30:31, then it means that the timestamp of the first log data and the timestamp corresponding to the previous log data belong to the same second; if the timestamp of the first log data is between 10:30:30 and 10:30:31, and the timestamp corresponding to the previous log data is between 10:30:29 and 10:30:30, then it means that the timestamp of the first log data and the timestamp corresponding to the previous log data do not belong to the same second.

[0111] If the timestamp of the first log data and the timestamp corresponding to the previous log data do not belong to the same second, execute the following steps 304 to 312; if the timestamp of the first log data and the timestamp corresponding to the previous log data belong to the same second, execute the following steps 306 to 312. If the timestamp of the first log data and the timestamp corresponding to the previous log data belong to the same second, the log counter may also be increased by 1.

[0112] 304. The log counter is cleared to 0.

[0113] The log counter can be used to record the total number of logs processed by the log processing process within 1 second.

[0114] In some embodiments, when the timestamp of the first log data and the timestamp corresponding to the previous log data do not belong to the same second, it means that the first log data is the first log data in the new 1 second, then the log counter can be cleared and added by 1 after clearing; and the count of the above log counter can be used as the total number of logs in the current new 1 second.

[0115] 305. Output the information currently recorded in the hash table.

[0116] The hash table may be a log key-value pair table, which may include multiple key-value pairs, each of which includes a keyword and a value corresponding to the keyword. In the log key-value pair table, the keyword of each key-value pair is used to represent log information of a log type, and the corresponding value corresponding to the keyword is the number of occurrences of log data of this log type within 1 second.

[0117] In an embodiment of the present application, when the timestamp of the first log data and the timestamp corresponding to the previous log data do not belong to the same second, it is possible to detect whether the hash table is empty. When the hash table is not empty, it is considered that the information recorded in the hash table is all information within the previous second, and the information recorded in the hash table can be output.

[0118] The above-mentioned step 305 is an optional step, that is, in the embodiment of the present application, step 305 can be executed when the timestamp of the first log data and the timestamp corresponding to the previous log data do not belong to the same second, or step 305 may not be executed when the timestamp of the first log data and the timestamp corresponding to the previous log data do not belong to the same second.

[0119] It should be noted that the information recorded in the output hash table can be stored in the hash table to facilitate subsequent analysis based on the information such as the log type recorded in the hash table within 1 second and the number of occurrences corresponding to each log type.

[0120] 306. Check whether the log counter exceeds the maximum number of log entries output per second.

[0121] When checking whether the log counter exceeds the maximum number of log entries output per second, the count of the log counter may be compared with a preset number threshold to determine whether the total number of logs is less than the preset number threshold. The maximum number of log entries is the preset number threshold.

[0122] When the log counter exceeds the maximum number of log entries output per second, the following steps 311 and 312 are executed; when the log counter does not exceed the maximum number of log entries output per second, the following steps 307 to 312 are executed.

[0123] 307. Check whether the corresponding log entry can be found in the hash table.

[0124] The corresponding log entry refers to a log entry of a target log type to which the first log data belongs.

[0125] Check whether the target keyword exists in the hash table to determine whether there is a corresponding log entry. The target keyword is used to represent the log information of the target log type, and the value corresponding to the target keyword is used to represent the number of occurrences of the log data of the target log type within a preset time period.

[0126] When the corresponding log entry can be found in the hash table, the following steps 308 to 312 can be executed; when the corresponding log entry cannot be found in the hash table, the following steps 309 to 312 can be executed.

[0127] 308. Check whether the value of the corresponding log entry in the hash table exceeds the maximum limit.

[0128] Among them, checking whether the value of the corresponding log entry in the hash table exceeds the maximum limit may be to determine whether the number of log occurrences is less than a preset number threshold. Among them, the maximum limit may refer to the preset number threshold.

[0129] When the value of the corresponding log entry in the hash table exceeds the maximum limit, the following steps 311 and 312 may be directly executed; when the value of the corresponding log entry in the hash table does not exceed the maximum limit, the following steps 309 and 310 may be executed.

[0130] 309. Insert / update the corresponding log entry in the hash table.

[0131] 310. In some embodiments, if the corresponding log entry can be found in the hash table, and the value of the corresponding log entry in the hash table does not exceed the maximum limit, it means that log data of the same log type as the first log data has appeared within 1 second, and after the first log data of the same log type appears again, the value corresponding to the target keyword can be increased by 1, that is, the number of occurrences of the log data of the target log type in the preset unit time is accumulated. Output the first log data.

[0132] Outputting the first log data may refer to storing the first log data in a log file and deleting the first log data from a log output queue.

[0133] In some embodiments, after the first log data is output, the log counter may be incremented by 1 to accumulate the total number of logs.

[0134] It should be noted that the log counter incremented by 1 in the description of step 303 and the log counter incremented by 1 in the description of step 310 are two different solutions and will not appear at the same time. The log counter incremented by 1 in the description of step 303 records the sum of the first quantity and the second quantity, wherein the first quantity is the quantity of log data processed by the log processing process and stored in the log file, and the second quantity is the quantity of log data processed by the log processing process and not stored in the log file.

[0135] In the description of the above step 310, the log counter plus 1 records the first quantity.

[0136] 311. Update the value of the corresponding log entry in the hash table.

[0137] If the corresponding log entry cannot be found in the hash table, it means that log data of the same log type as the first log data has not appeared within 1 second. Then, after the first log data of the same log type appears for the first time, a target keyword can be added to the log key-value pair table, and the value corresponding to the target keyword is increased by 1, which means that the number of occurrences of log data of the target log type in the preset unit time length can be counted.

[0138] 312. The head node generates a log output queue.

[0139] The head node outputs the log output queue, that is, deletes the first log data from the log output queue, and does not store the first log data in the log file.

[0140] In the above embodiment, the total number of logs and the number of log occurrences of the same type of logs within a preset unit time are counted respectively through a log counter and a hash table, and the flow of log data is limited from these two levels to avoid a large number of redundant logs occupying the disk space corresponding to the log files, and to a certain extent avoid the effective logs from being flushed.

[0141] Figure 4 A schematic diagram of the structure of a log data processing device provided in an embodiment of the present application.

[0142] Corresponding to the above method embodiment, the present application embodiment also provides a log data processing device, see Figure 4 , the log data processing device 400 includes:

[0143] The data acquisition module 401 is used to obtain the first log data corresponding to the head node from the current log output queue, and determine the target log type of the first log data; obtain the total number of logs processed by the log processing process within a preset unit time; obtain the number of log occurrences of the target log type within the preset unit time;

[0144] The data processing module 402 is used to store the first log data in a log file and delete the first log data from the log output queue when the total number of logs is less than a preset number threshold and the number of log occurrences is less than a preset number threshold.

[0145] Optionally, the data processing module 402 is further used to: when the total number of logs is greater than or equal to a preset number threshold, or the number of log occurrences is greater than or equal to a preset number threshold, delete the first log data from the log output queue.

[0146] Optionally, the data acquisition module 401 is further used to: before acquiring the total number of logs processed by the log processing process within a preset unit time, acquire the timestamp of the first log data from the log output queue; acquire the timestamp corresponding to the previous log data of the first log data;

[0147] The data acquisition module 402 is specifically used for: obtaining the total number of logs processed by the log processing process within a preset unit time length, including: when the timestamp of the first log data and the timestamp corresponding to the previous log data are within a preset time period, controlling the log counter to add 1, the log counter being used to record the total number of logs, and the duration of the preset time period is the preset unit time length; when the timestamp of the first log data and the timestamp corresponding to the previous log data are not within the preset time period, clearing the log counter and then adding 1; and using the count of the log counter as the total number of logs.

[0148] Optionally, the total number of logs is the number of logs processed by the log processing process and stored in the log file within the preset unit time.

[0149] Optionally, the data acquisition module 401 is specifically used to: the acquisition of the number of log occurrences of the target log type within the preset unit time includes: when the target keyword exists in the log key-value pair table, taking the value corresponding to the target keyword as the number of log occurrences; when the target keyword does not exist in the log key-value pair table, determining that the number of log occurrences is 0;

[0150] Among them, the log key-value pair table includes multiple groups of key-value pairs, each group of key-value pairs includes a keyword and a value corresponding to the keyword, the target keyword is used to represent the log information of the target log type, and the value corresponding to the target keyword is used to represent the number of occurrences of the log data of the target log type within a preset time period.

[0151] Optionally, the data processing module 402 is further configured to:

[0152] In the case where the target keyword exists in the log key-value pair table, after taking the value corresponding to the target keyword as the number of occurrences of the log, the method further comprises: in the case where the target keyword exists in the log key-value pair table, adding 1 to the value corresponding to the target keyword;

[0153] or,

[0154] When the target keyword does not exist in the log key-value pair table, after determining that the log occurrence count is 0, the method further includes: when the target keyword does not exist in the log key-value pair table, adding the target keyword to the log key-value pair table and adding 1 to the value corresponding to the target keyword.

[0155] Optionally, the device also includes: a log queue processing module 403; the log queue processing module 403 is used to, before the data acquisition module 401 obtains the first log data corresponding to the head node from the current log output queue, when the log data in the first log queue has been processed by the log processing thread, and there is log data written by the business thread in the second log queue, use the cleared first log queue as the current log write queue, and use the second log queue as the current log output queue, the first log queue is the original log output queue, and the second log queue is the original log write queue.

[0156] Optionally, the log queue processing module 403 is also used to set a mutex lock for the second log queue before obtaining the first log data corresponding to the head node from the current log output queue, when the log data in the first log queue have been processed by the log processing thread and there is log data written by the business thread in the second log queue, and the mutex lock is used to deny the business thread from accessing the second log queue.

[0157] The specific details of each module or unit in the above device have been described in detail in the corresponding method, so they will not be repeated here.

[0158] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0159] In an exemplary embodiment of the present application, an electronic device is further provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the log data processing method described above in this exemplary embodiment.

[0160] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. It should be noted that: Figure 5 The electronic device 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0161] like Figure 5 As shown, electronic device 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In RAM 503, various programs and data required for system operation are also stored. Central processing unit 501, ROM 502 and RAM 503 are connected to each other via a bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0162] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a local area network (LAN) card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0163] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for the above-mentioned log data processing method. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit 501, various functions defined in the device of the present application are executed.

[0164] In an embodiment of the present application, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned log data processing method is implemented.

[0165] It should be noted that the computer-readable storage medium shown in the present application may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency, etc., or any suitable combination of the above.

[0166] In an embodiment of the present application, a computer program product is also provided. When the computer program product is run on a computer, the computer executes the above-mentioned log processing method.

[0167] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0168] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A log data processing method, characterized in that: The method comprises: Obtaining first log data corresponding to the head node from the current log output queue, and determining a target log type of the first log data; Get the total number of logs processed by the log processing process within the preset unit time; Obtain the number of occurrences of the target log type within the preset unit time; When the total number of logs is less than a preset number threshold, and the number of occurrences of the logs is less than a preset number threshold, the first log data is stored in a log file, and the first log data is deleted from the log output queue.

2. The method according to claim 1, characterized in that Before obtaining the total number of logs processed by the log processing process within a preset unit time, the method further includes: Obtaining a timestamp of the first log data from the log output queue; Obtaining a timestamp corresponding to a previous log data of the first log data; The obtaining of the total number of logs processed by the log processing process within a preset unit time includes: If the timestamp of the first log data and the timestamp corresponding to the previous log data are within a preset time period, then control a log counter to add 1, the log counter is used to record the total number of logs, and the duration of the preset time period is the preset unit duration; If the timestamp of the first log data and the timestamp corresponding to the previous log data are not within the preset time period, clearing the log counter and then adding 1; The count of the log counter is used as the total number of logs.

3. The method according to claim 1, characterized in that The obtaining the number of occurrences of the log of the target log type within the preset unit time includes: If the target keyword exists in the log key-value pair table, the value corresponding to the target keyword is used as the number of occurrences of the log; If the target keyword does not exist in the log key-value pair table, determining that the number of occurrences of the log is 0; Among them, the log key-value pair table includes multiple groups of key-value pairs, each group of key-value pairs includes a keyword and a value corresponding to the keyword, the target keyword is used to represent the log information of the target log type, and the value corresponding to the target keyword is used to represent the number of occurrences of the log data of the target log type within a preset time period.

4. The method according to claim 3, characterized in that: In the case where the target keyword exists in the log key-value pair table, after taking the value corresponding to the target keyword as the number of occurrences of the log, the method further comprises: in the case where the target keyword exists in the log key-value pair table, adding 1 to the value corresponding to the target keyword; or, When the target keyword does not exist in the log key-value pair table, after determining that the log occurrence count is 0, the method further includes: when the target keyword does not exist in the log key-value pair table, adding the target keyword to the log key-value pair table and adding 1 to the value corresponding to the target keyword.

5. The method according to claim 1, characterized in that Before obtaining the first log data corresponding to the head node from the current log output queue, the method further includes: When all log data in the first log queue have been processed by the log processing thread and there is log data written by the business thread in the second log queue, the cleared first log queue will be used as the current log write queue, and the second log queue will be used as the current log output queue. The first log queue is the original log output queue, and the second log queue is the original log write queue.

6. The method according to claim 5, characterized in that Before obtaining the first log data corresponding to the head node from the current log output queue, the method further includes: When all log data in the first log queue have been processed by the log processing thread and there is log data written by the business thread in the second log queue, a mutex lock is set for the second log queue, and the mutex lock is used to deny the business thread access to the second log queue.

7. The method according to claim 1, characterized in that: The method further comprises: When the total number of logs is greater than or equal to a preset number threshold, or the number of occurrences of the log is greater than or equal to a preset number threshold, the first log data is deleted from the log output queue, and the first log data is not stored in the log file.

8. A log data processing device, characterized in that: The device comprises: A data acquisition module, used to acquire first log data corresponding to the head node from the current log output queue, and determine a target log type of the first log data; Get the total number of logs processed by the log processing process within the preset unit time; Obtain the number of occurrences of the target log type within the preset unit time; The data processing module is used to store the first log data in a log file and delete the first log data from the log output queue when the total number of logs is less than a preset number threshold and the number of log occurrences is less than a preset number threshold.

9. An electronic device, characterized in that: include: A processor, wherein the processor is used to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the log data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the log data processing method according to any one of claims 1 to 7 is implemented.

Images