Log pattern recognition method and system based on time window aggregation

Abstract

The invention discloses a log pattern recognition method and system based on time window aggregation, and the method comprises the steps: dividing time windows according to a preset time interval, and aggregating logs in the same time window; performing word segmentation processing on the log text, and segmenting the log text into elements; judging whether the element is a variable or a fixed text, and replacing the variable in the element with a fixed character; constructing a primary log mode according to the elements; traversing all the logs in each time window, and merging the logs with the same primary log mode to obtain a primary log mode set in each time window. According to the technical scheme provided by the invention, the log modes of the logs are identified and merged according to the divided time windows, so that the identification accuracy can be improved; the log modes are subjected to secondary combination, so that the recognition accuracy is improved; during re-running after parameter adjustment of the algorithm, only second-time merging can be carried out, and the running speed of re-running can be increased; high-parallelism calculation is supported, and the processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of log data processing, and in particular to a log pattern recognition method and system based on time window aggregation. Background Art

[0002] With the development of IT technology, the operational stability of IT systems is becoming more and more important. Operation and maintenance personnel need to monitor the operating status of IT systems in real time, and log data is an important monitoring object.

[0003] Traditional log monitoring solutions are configured based on keyword rules. When a log keyword appears more than the set threshold in a unit of time, an alarm is issued. This solution has high configuration costs and is difficult to apply and maintain on a large scale.

[0004] In recent years, log anomaly detection solutions that do not rely on rules have gradually been applied. The core is to read logs in real time, identify log patterns, and detect whether the current number of each log pattern exceeds the historical baseline, so as to achieve monitoring and alarm. Compared with keyword rule solutions, log anomaly detection greatly reduces the cost of manual configuration. The key step of log anomaly detection is log pattern recognition, and the effect of log pattern recognition will directly affect the overall effect of log anomaly detection.

[0005] The principle of the log pattern recognition algorithm is to identify the fixed text and dynamic variables in the log data and classify them into different log patterns. For example, a program prints a log statement, print(f"user {user_id} loginfail"), where user_id is a variable in the program execution; if the generated log data is user 1000123 loginfail, then 1000123 is a variable, which is converted to * to get the log pattern user * login fail. The implementation plan of log anomaly detection is to analyze the log pattern of each log, find out which logs belong to the same log pattern, count the number of occurrences of each log pattern per unit time, and output it to the anomaly detection algorithm for subsequent processing.

[0006] Common log pattern recognition algorithms include the DeepLog algorithm based on deep learning, the Drain algorithm based on a tree structure, and the Spell algorithm based on a finite state machine. These algorithms have the following disadvantages: 1. Low throughput. Most algorithms match logs one by one with all current log patterns. The amount of calculation is too large, resulting in high requirements for server hardware and difficulty in large-scale practical application.

[0007] 2. The recognition error rate is high. Many algorithms make too ideal assumptions about the log format and can only process logs with a uniform format. When processing logs with complex and changing formats, pattern recognition for each log is prone to recognition errors, resulting in false alarms in subsequent monitoring.

[0008] 3. The computing parallelism is low. The algorithm cannot support computing parallelism of more than 16 processes. As a result, when the log volume of a single log source is high, the algorithm cannot meet the real-time processing requirements.

[0009] 4. The rerun speed is slow. When the algorithm parameters are tuned, historical data needs to be rerun for retraining. Most algorithms do not support the use of previous run results to speed up reruns, resulting in the need to recalculate all log data every time a rerun is performed, which often takes several hours or even days and is very time-consuming. Summary of the invention

[0010] In view of the above-mentioned deficiencies in the current technology, the present invention provides a log pattern recognition method based on time window aggregation, which recognizes and merges log patterns according to the divided time windows of log data, which can greatly improve the recognition accuracy and processing efficiency.

[0011] To achieve the above object, the embodiments of the present invention adopt the following technical solutions: A log pattern recognition method based on time window aggregation includes the following steps: Divide the time window according to the preset time interval and aggregate the logs in the same time window; Perform word segmentation on the log text and divide the log text into elements; Determine whether the element is a variable or fixed text, and replace the variable in the element with fixed characters; Build a primary log schema from the elements; All logs in each time window are traversed, and logs with the same primary log pattern are merged to obtain the primary log pattern set in each time window.

[0012] According to one aspect of the present invention, the word segmentation processing of the log text to divide the log text into elements includes: Treat each log as a string and perform text segmentation. The segmentation method is to use all English and Chinese symbols as delimiters, each delimiter itself and the words before and after the delimiter as elements, and separate all Chinese, English, and numbers to ensure that each element contains only one of the four types of Chinese, English, numbers, and Chinese and English symbols.

[0013] According to one aspect of the present invention, the determining element is a variable or a fixed text comprises: Count the number of logs N in a single time window; Count the number of occurrences of each element C; Set the weights W of different elements; Calculate the value of N*W; Compare C and N*W. When C>N*W, the element is judged as a variable, otherwise the element is judged as fixed text.

[0014] According to one aspect of the present invention, constructing a primary log pattern according to elements includes: replacing variables in the elements with fixed characters, and the replaced elements are the primary log pattern.

[0015] According to one aspect of the present invention, the log pattern recognition method based on time window aggregation further includes: According to the primary log pattern set in each time window, the number of occurrences of each primary log pattern in a single time window is obtained; Log anomaly detection is performed based on the number of occurrences of each primary log pattern in a single time window.

[0016] According to one aspect of the present invention, the log pattern recognition method based on time window aggregation further includes: According to the primary log mode, build a historical log mode list; The primary log pattern of the logs in the same time window is compared with the historical log pattern for similarity, and the primary log pattern is incorporated into the historical log pattern list according to the similarity.

[0017] According to one aspect of the present invention, comparing the similarity between the primary log pattern and the historical log pattern of the logs in the same time window, and incorporating the primary log pattern into the historical log pattern list according to the similarity comprises: The historical log pattern list is [p1, p2, ..., pM], with a total of M log patterns. The initial state of the historical log pattern list is an empty list; Set the similarity threshold T; Remove all stop characters from the primary log pattern and log pattern to form a short pattern; In chronological order, traverse each primary log pattern in each window in turn, compare the short pattern formed by each primary log pattern with the short patterns formed by all M log patterns one by one, and obtain the similarity; The highest similarity is selected. If the highest similarity reaches the threshold T, the primary log pattern is merged into the log pattern corresponding to the short pattern with the highest similarity. Otherwise, if the highest similarity does not reach the threshold T, the primary log pattern becomes an independent new log pattern and is added to the historical log pattern list.

[0018] According to one aspect of the present invention, the short pattern formed by each primary log pattern is compared with the short patterns formed by all M log patterns one by one for similarity, and obtaining the similarity includes: Calculate the longest common subsequence length L between two short patterns; Get the larger value Z of the string lengths of the two short patterns; Similarity S = L / Z.

[0019] According to one aspect of the present invention, the log pattern recognition method based on time window aggregation further includes: According to the historical log pattern list, obtain the number of occurrences of each log pattern in a single time window; Log anomaly detection is performed based on the number of occurrences of each log pattern in a single time window.

[0020] According to one aspect of the present invention, the log pattern recognition method based on time window aggregation further includes: When adjusting the parameters in the method and re-running it, you only need to re-construct the historical log pattern list, calculate the similarity, and merge the primary log pattern into the historical log pattern list according to the similarity. You do not need to re-construct the primary log pattern and obtain the primary log pattern set.

[0021] According to one aspect of the present invention, the log pattern recognition method based on time window aggregation further includes: Parallel processing can be used to process logs in different time windows at the same time.

[0022] A log pattern recognition system based on time window aggregation, based on the log pattern recognition method based on time window aggregation as described above, includes: The time window division module is used to divide the time window according to the preset time interval and aggregate the logs in the same time window; The word segmentation module is used to perform word segmentation on the log text and divide the log text into elements; The variable processing module is used to determine whether an element is a variable or a fixed text, and replace the variable in the element with a fixed character; A primary log pattern building module, used to build a primary log pattern based on elements; The primary log pattern merging module is used to traverse all logs in each time window, merge logs with the same primary log pattern, and obtain the primary log pattern set in each time window.

[0023] According to one aspect of the present invention, the log pattern recognition system based on time window aggregation further includes: A primary counting module, used to obtain the number of occurrences of each primary log pattern in a single time window according to the primary log pattern set in each time window; The primary anomaly detection module is used to perform log anomaly detection based on the number of occurrences of each primary log pattern in a single time window; A historical pattern building module is used to build a historical log pattern list according to the primary log pattern; A similarity comparison module is used to compare the similarity between the primary log pattern and the historical log pattern of the log in the same time window, and merge the primary log pattern into the historical log pattern list according to the similarity; The counting module is used to obtain the number of occurrences of each log pattern in a single time window based on the historical log pattern list; The anomaly detection module is used to perform log anomaly detection based on the number of occurrences of each log pattern in a single time window.

[0024] Advantages of the present invention: The present invention provides a log pattern recognition method based on time window aggregation, which recognizes and merges log patterns according to divided time windows of log data, thereby improving recognition accuracy.

[0025] The method provided by the present invention can perform a secondary merger of log patterns based on a time window, which can improve processing throughput, reduce the probability of log pattern recognition errors, and improve recognition accuracy compared to existing solutions. When the algorithm is re-run after parameter adjustment, only the second merger can be performed, which can improve the running speed of the re-run.

[0026] The method provided by the present invention can support high-parallel computing and improve processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 This is a flow chart of a log pattern recognition method based on time window aggregation according to Embodiment 1 of the present invention; Figure 2 This is a flow chart of a log pattern recognition method based on time window aggregation described in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Embodiment 1 like Figure 1 As shown, a log pattern recognition method based on time window aggregation includes the following steps: S1: Divide the time window into preset time intervals and aggregate the logs in the same time window.

[0031] The preset time interval may be 1 minute.

[0032] After reading the logs, logs within the same minute are divided into the same time window according to the time information of the logs, and logs in different minutes are divided into different time windows.

[0033] S2: Perform word segmentation on the log text and divide the log text into elements.

[0034] The word segmentation processing of the log text and the segmentation of the log text into elements include: Treat each log as a string and perform text segmentation. The segmentation method is to use all English and Chinese symbols as delimiters, each delimiter itself and the words before and after the delimiter as elements, and separate all Chinese, English, and numbers to ensure that each element contains only one of the four types of Chinese, English, numbers, and Chinese and English symbols.

[0035] Elements can also be called tokens.

[0036] S3: Determine whether the element is a variable or a fixed text, and replace the variable in the element with a fixed character.

[0037] Assume that the log data in the window is [log1, log2, ..., logN], with a total of N logs, numbered 1..N respectively.

[0038] The judgment element is a variable or a fixed text including: S31: Count the number of logs N in a single time window.

[0039] S32: Count the number of occurrences C of each element.

[0040] S33: Set the weights W of different elements.

[0041] The design concept of weight W is that the more likely an element is to become a variable, the higher its weight is, and the weight is as small as possible within a reasonable range, so as to minimize the probability of non-variables being mistakenly judged as variables. That is, the probability that different log patterns are mistakenly merged into the same primary log pattern is very low, while the probability that the same log pattern is mistakenly merged into different log patterns is high.

[0042] The weight W can be set as follows: when the element is a number, W is 0.4; when the element is English, W=0.04; when the element is Chinese, W=0.02; when the element is Chinese and English symbols, W=0.01.

[0043] S34: Calculate the value of N*W.

[0044] S35: Compare C and N*W. When C>N*W, the element is determined to be a variable, otherwise the element is determined to be a fixed text.

[0045] When replacing variables in an element with fixed characters, the fixed characters can be set to * or other characters without specific meanings.

[0046] S4: Construct primary log schema based on elements.

[0047] The constructing of the primary log pattern according to the element includes: replacing the variables in the element with fixed characters, and the replaced element is the primary log pattern.

[0048] Traverse each element of each log. If it is determined to be a variable, replace it with *. If it is determined to be a fixed text, no replacement operation is performed. The replaced data is called the primary log pattern, abbreviated as pp.

[0049] S5: traverse all logs in each time window, merge the logs with the same primary log pattern, and obtain the primary log pattern set in each time window.

[0050] Traverse each log, if the primary log pattern of a log is the same as the primary log pattern of the previous log, merge them together, and record the sequence number of each log after merging. Perform the above calculation on the log data of each window to obtain the primary log pattern set of each window. The calculation results of each window do not affect each other, and multiple processes can be used for parallel calculation.

[0051] Preferably, the log pattern recognition method based on time window aggregation further includes: Parallel processing can be used to process logs in different time windows at the same time.

[0052] Because the logs of a time window are subjected to independent pattern recognition, that is, when merged, the pattern recognition of different time windows does not affect each other, the logs of different time windows can be assigned to different processes for processing, thereby greatly improving the computational parallelism.

[0053] Preferably, the log pattern recognition method based on time window aggregation further includes: S6: According to the primary log pattern sets in each time window, obtain the number of occurrences of each primary log pattern in a single time window.

[0054] S7: Perform log anomaly detection according to the number of occurrences of each primary log pattern in a single time window.

[0055] The beneficial effects of this embodiment are: The method can improve the accuracy of identification by identifying and merging log patterns according to the divided time windows of log data. The method can also improve the processing efficiency by utilizing parallel computing.

[0056] This method is a pattern recognition algorithm within a single time window. It performs an overall analysis of all logs within the window. After the logs are segmented, the variables are converted to * through a variable recognition algorithm based on word frequency, dependency, and structure. Finally, logs with the same content are merged together, which is called the first merge. After the merger, a log pattern is formed.

[0057] Two types of errors may occur during pattern recognition. The first error is that a word that is actually a variable is recognized as a non-variable by the algorithm, which will cause multiple logs with the same pattern to be recognized as different patterns. The second error is that a fixed text that is actually a non-variable is mistakenly recognized as a variable, causing multiple logs with different log patterns to be mistakenly recognized as the same pattern.

[0058] The method of this embodiment uses conservative variable identification algorithm parameters during processing, which can minimize the above-mentioned second error.

[0059] Embodiment 2 like Figure 2 As shown, a log pattern recognition method based on time window aggregation includes the following steps: S1: Divide the time window into preset time intervals and aggregate the logs in the same time window.

[0060] The preset time interval may be 1 minute.

[0061] After reading the logs, logs within the same minute are divided into the same time window according to the time information of the logs, and logs in different minutes are divided into different time windows.

[0062] S2: Perform word segmentation on the log text and divide the log text into elements.

[0063] The word segmentation processing of the log text and the segmentation of the log text into elements include: Treat each log as a string and perform text segmentation. The segmentation method is to use all English and Chinese symbols as delimiters, each delimiter itself and the words before and after the delimiter as elements, and separate all Chinese, English, and numbers to ensure that each element contains only one of the four types of Chinese, English, numbers, and Chinese and English symbols.

[0064] Elements can also be called tokens.

[0065] S3: Determine whether the element is a variable or a fixed text, and replace the variable in the element with a fixed character.

[0066] Assume that the log data in the window is [log1, log2, ..., logN], with a total of N logs, numbered 1..N respectively.

[0067] The judgment element is a variable or a fixed text including: S31: Count the number of logs N in a single time window.

[0068] S32: Count the number of occurrences C of each element.

[0069] S33: Set the weights W of different elements.

[0070] The design concept of weight W is that the more likely an element is to become a variable, the higher its weight is, and the weight is as small as possible within a reasonable range, so as to minimize the probability of non-variables being mistakenly judged as variables. That is, the probability that different log patterns are mistakenly merged into the same primary log pattern is very low, while the probability that the same log pattern is mistakenly merged into different log patterns is high.

[0071] The weight W can be set as follows: when the element is a number, W is 0.4; when the element is English, W=0.04; when the element is Chinese, W=0.02; when the element is Chinese and English symbols, W=0.01.

[0072] S34: Calculate the value of N*W.

[0073] S35: Compare C and N*W. When C>N*W, the element is determined to be a variable, otherwise the element is determined to be a fixed text.

[0074] When replacing variables in an element with fixed characters, the fixed characters can be set to * or other characters without specific meanings.

[0075] S4: Construct primary log schema based on elements.

[0076] The constructing of the primary log pattern according to the element includes: replacing the variables in the element with fixed characters, and the replaced element is the primary log pattern.

[0077] Traverse each element of each log. If it is determined to be a variable, replace it with *. If it is determined to be a fixed text, no replacement operation is performed. The replaced data is called the primary log pattern, abbreviated as pp.

[0078] S5: traverse all logs in each time window, merge the logs with the same primary log pattern, and obtain the primary log pattern set in each time window.

[0079] Traverse each log, if the primary log pattern of a log is the same as the primary log pattern of the previous log, merge them together, and record the sequence number of each log after merging. Perform the above calculation on the log data of each window to obtain the primary log pattern set of each window. The calculation results of each window do not affect each other, and multiple processes can be used for parallel calculation.

[0080] Step S5 is the first merging performed by this method.

[0081] S6: Construct a historical log pattern list according to the primary log pattern.

[0082] S7: Compare the similarity between the primary log pattern and the historical log pattern of the logs in the same time window, and merge the primary log pattern into the historical log pattern list according to the similarity.

[0083] The step S7 of comparing the similarity between the primary log pattern and the historical log pattern of the log in the same time window and incorporating the primary log pattern into the historical log pattern list according to the similarity includes: S71: The historical log mode list is [p1, p2, ..., pM], with a total of M log modes. The initial state of the historical log mode list is an empty list.

[0084] S72: Set a similarity threshold T.

[0085] The threshold T is an adjustable parameter and its default value is 0.85.

[0086] S73: All stop characters in the primary log pattern and the log pattern are removed to form a short pattern.

[0087] S74: traverse each primary log pattern in each window in chronological order, compare the short pattern formed by each primary log pattern with the short patterns formed by all the M log patterns one by one to obtain similarity.

[0088] The short pattern formed by each primary log pattern is compared with the short patterns formed by all the M log patterns one by one for similarity, and obtaining the similarity comprises: Calculate the longest common subsequence length L between two short patterns; Get the larger value Z of the string lengths of the two short patterns; Similarity S = L / Z.

[0089] S75: Select the highest similarity. If the highest similarity reaches the threshold T, merge the primary log pattern into the log pattern corresponding to the short pattern with the highest similarity. Otherwise, if the highest similarity does not reach the threshold T, the primary log pattern becomes an independent new log pattern and is added to the historical log pattern list.

[0090] Step S7 is the second merging performed by this method.

[0091] In the first merge, the error situation of "actually the same log pattern is mistakenly merged into different primary log patterns" may occur. This error situation will be effectively alleviated in the second merge. This is because these primary log patterns will be merged into the same log pattern in the historical log pattern list with a high probability, thereby reducing the error rate of the final log pattern recognition.

[0092] Preferably, the log pattern recognition method based on time window aggregation further includes: S8: According to the historical log pattern list, obtain the number of occurrences of each log pattern in a single time window.

[0093] In the second merge, the primary log patterns are merged into log patterns. When a short pattern is merged into a historical log pattern, the log sequence number information it contains will also be merged. After all primary log patterns in a time window are merged, the sequence number information corresponding to each log pattern in the time window can be obtained. By counting the number of sequence numbers, the number of occurrences of each log pattern in the time window can be obtained.

[0094] S9: Perform log anomaly detection based on the number of occurrences of each log pattern in a single time window.

[0095] After traversing each time window, the number of occurrences of each log pattern in the window is output to the downstream timing anomaly detection module, which can analyze whether the number exceeds the historical normal range. If so, an abnormal alarm is reported, thereby realizing log anomaly detection.

[0096] Preferably, the log pattern recognition method based on time window aggregation further includes: When adjusting the parameters in the method and re-running it, you only need to re-construct the historical log pattern list, calculate the similarity, and merge the primary log pattern into the historical log pattern list according to the similarity. You do not need to re-construct the primary log pattern and obtain the primary log pattern set.

[0097] The algorithm parameters used in this method for the first merge are conservative and do not need to be tuned in most cases. What needs to be tuned are the algorithm parameters for the second merge. That is, when rerunning, there is no need to rerun the steps of the first merge. The historical results of the first merge can be used directly, and only the steps of the second merge need to be rerun. Since the amount of calculation for the first merge is much higher than that for the second merge, it is equivalent to that most calculations do not need to be repeated, and only the second merge with a small amount of calculation needs to be rerun, which greatly improves the rerun speed.

[0098] Preferably, the log pattern recognition method based on time window aggregation further includes: Parallel processing can be used to process logs in different time windows at the same time.

[0099] Because the logs of a time window are subjected to independent pattern recognition, that is, the pattern recognitions of different time windows do not affect each other during the first merge, the logs of different time windows can be assigned to different processes for processing, thereby greatly improving the computational parallelism.

[0100] The beneficial effects of this embodiment are: This method divides the log data according to time windows, and performs a first identification and merging of independent log patterns for the logs in each time window; then, the log patterns in multiple windows are secondarily identified and merged in chronological order.

[0101] This method is a pattern recognition algorithm within a single time window. It performs an overall analysis of all logs within the window. After the logs are segmented, the variables are converted to * through a variable recognition algorithm based on word frequency, dependency, and structure. Finally, logs with the same content are merged together, which is called the first merge. After the merger, a log pattern is formed.

[0102] The second merge of this method is to merge windows one by one in chronological order, and match all log patterns of the current window with all historical log patterns for similarity. If the similarity is too low, a new log pattern is formed, otherwise it is merged into the corresponding historical log pattern. The purpose of the second merge is to avoid matching each log with all current log patterns. By taking advantage of the fact that the number of log patterns is far less than the number of logs, the number of matches is greatly reduced compared to the log-by-log matching strategy of traditional algorithms, thereby improving throughput. The similarity matching algorithm of the second merge corrects the pattern recognition errors of the first merge, thereby achieving adaptation to multiple types.

[0103] Two types of errors may occur during pattern recognition. The first error is that a word that is actually a variable is recognized as a non-variable by the algorithm, which will cause multiple logs with the same pattern to be recognized as different patterns. The second error is that a fixed text that is actually a non-variable is mistakenly recognized as a variable, causing multiple logs with different log patterns to be mistakenly recognized as the same pattern.

[0104] The method of this embodiment uses conservative variable recognition algorithm parameters when performing the first merging process, which can reduce the second error as much as possible. The first error will be corrected in the second merging. Through two mergings, the accuracy of recognition can be greatly improved.

[0105] This method can also improve processing efficiency by utilizing parallel computing.

[0106] Embodiment 3 A log pattern recognition system based on time window aggregation, based on the log pattern recognition method based on time window aggregation described in Embodiment 1, includes: The time window division module is used to divide the time window according to the preset time interval and aggregate the logs in the same time window; The word segmentation module is used to perform word segmentation on the log text and divide the log text into elements; The variable processing module is used to determine whether an element is a variable or a fixed text, and replace the variable in the element with a fixed character; A primary log pattern building module, used to build a primary log pattern based on elements; The primary log pattern merging module is used to traverse all logs in each time window, merge logs with the same primary log pattern, and obtain the primary log pattern set in each time window.

[0107] Preferably, the log pattern recognition system based on time window aggregation further includes: A primary counting module, used to obtain the number of occurrences of each primary log pattern in a single time window according to the primary log pattern set in each time window; The primary anomaly detection module is used to perform log anomaly detection based on the number of occurrences of each primary log pattern in a single time window.

[0108] Embodiment 4 A log pattern recognition system based on time window aggregation, based on the log pattern recognition method based on time window aggregation described in Embodiment 2, includes: The time window division module is used to divide the time window according to the preset time interval and aggregate the logs in the same time window; The word segmentation module is used to perform word segmentation on the log text and divide the log text into elements; The variable processing module is used to determine whether an element is a variable or a fixed text, and replace the variable in the element with a fixed character; A primary log pattern building module, used to build a primary log pattern based on elements; The primary log pattern merging module is used to traverse all logs in each time window, merge the logs with the same primary log pattern, and obtain the primary log pattern set in each time window; A historical pattern building module is used to build a historical log pattern list according to the primary log pattern; The similarity comparison module is used to compare the similarity between the primary log pattern and the historical log pattern of the log in the same time window, and merge the primary log pattern into the historical log pattern list according to the similarity.

[0109] Preferably, the log pattern recognition system based on time window aggregation further includes: The counting module is used to obtain the number of occurrences of each log pattern in a single time window based on the historical log pattern list; The anomaly detection module is used to perform log anomaly detection based on the number of occurrences of each log pattern in a single time window.

[0110] Embodiment 5 A computer program, when executed, implements the steps of the log pattern recognition method based on time window aggregation as described in embodiment one or two.

[0111] Embodiment 6 A readable storage medium stores a computer program as described in Example 5, and when the computer program is executed, the steps of the log pattern recognition method based on time window aggregation as described in Example 1 or 2 are implemented.

[0112] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A log pattern recognition method based on time window aggregation, characterized in that: The following steps are involved: Divide the time window according to the preset time interval and aggregate the logs in the same time window; Perform word segmentation on the log text and divide the log text into elements; Determine whether the element is a variable or fixed text, and replace the variable in the element with fixed characters; Build a primary log schema from the elements; Traverse all logs in each time window, merge logs with the same primary log pattern, and obtain the primary log pattern set in each time window; The log pattern recognition method based on time window aggregation also includes: According to the primary log mode, build a historical log mode list; Compare the similarity between the primary log pattern and the historical log pattern of the logs in the same time window, and merge the primary log pattern into the historical log pattern list according to the similarity; The similarity is calculated in the following way: Calculate the longest common subsequence length L between two log patterns; Get the larger value Z of the string lengths of the two log patterns; Similarity S = L / Z.

2. The log pattern recognition method based on time window aggregation according to claim 1 is characterized in that: The word segmentation processing of the log text and the segmentation of the log text into elements include: Treat each log as a string and perform text segmentation. The segmentation method is to use all English and Chinese symbols as delimiters, each delimiter itself and the words before and after the delimiter as elements, and separate all Chinese, English, and numbers to ensure that each element contains only one of the four types of Chinese, English, numbers, and Chinese and English symbols.

3. The log pattern recognition method based on time window aggregation according to claim 1 is characterized in that: The judgment element is a variable or a fixed text including: Count the number of logs N in a single time window; Count the number of occurrences of each element C; Set the weights W of different elements; Calculate the value of N*W; Compare C and N*W. When C>N*W, the element is judged as a variable, otherwise the element is judged as fixed text.

4. The log pattern recognition method based on time window aggregation according to claim 1 is characterized in that: The constructing of the primary log pattern according to the element includes: replacing the variables in the element with fixed characters, and the replaced element is the primary log pattern.

5. The log pattern recognition method based on time window aggregation according to claim 1 is characterized in that: The log pattern recognition method based on time window aggregation also includes: According to the primary log pattern set in each time window, the number of occurrences of each primary log pattern in a single time window is obtained; Log anomaly detection is performed based on the number of occurrences of each primary log pattern in a single time window.

6. The log pattern recognition method based on time window aggregation according to claim 1 is characterized in that: The comparing the similarity between the primary log pattern and the historical log pattern of the logs in the same time window and incorporating the primary log pattern into the historical log pattern list according to the similarity comprises: The historical log pattern list is [p1, p2, ..., pM], with a total of M log patterns. The initial state of the historical log pattern list is an empty list; Set the similarity threshold T; Remove all stop characters from the primary log pattern and log pattern to form a short pattern; In chronological order, traverse each primary log pattern in each window in turn, compare the short pattern formed by each primary log pattern with the short patterns formed by all M log patterns one by one, and obtain the similarity; The highest similarity is selected. If the highest similarity reaches the threshold T, the primary log pattern is merged into the log pattern corresponding to the short pattern with the highest similarity. Otherwise, if the highest similarity does not reach the threshold T, the primary log pattern becomes an independent new log pattern and is added to the historical log pattern list.

7. The log pattern recognition method based on time window aggregation according to claim 6 is characterized in that: The short pattern formed by each primary log pattern is compared with the short patterns formed by all the M log patterns one by one for similarity, and obtaining the similarity comprises: Calculate the longest common subsequence length L between two short patterns; Get the larger value Z of the string lengths of the two short patterns; Similarity S = L / Z.

8. The log pattern recognition method based on time window aggregation according to claim 1 is characterized in that: The log pattern recognition method based on time window aggregation also includes: According to the historical log pattern list, obtain the number of occurrences of each log pattern in a single time window; Log anomaly detection is performed based on the number of occurrences of each log pattern in a single time window.

9. The log pattern recognition method based on time window aggregation according to claim 1 is characterized in that: The log pattern recognition method based on time window aggregation also includes: When adjusting the parameters in the method and re-running it, you only need to re-construct the historical log pattern list, calculate the similarity, and merge the primary log pattern into the historical log pattern list according to the similarity. You do not need to re-construct the primary log pattern and obtain the primary log pattern set.

10. The log pattern recognition method based on time window aggregation according to claim 1 is characterized in that: The log pattern recognition method based on time window aggregation also includes: Parallel processing is used to process logs in different time windows simultaneously.

11. A log pattern recognition system based on time window aggregation, characterized in that: Applying the log pattern recognition method based on time window aggregation according to any one of claims 1 to 10, comprising: The time window division module is used to divide the time window according to the preset time interval and aggregate the logs in the same time window; The word segmentation module is used to perform word segmentation on the log text and divide the log text into elements; The variable processing module is used to determine whether an element is a variable or a fixed text, and replace the variable in the element with a fixed character; A primary log pattern building module, used to build a primary log pattern based on elements; The primary log pattern merging module is used to traverse all logs in each time window, merge logs with the same primary log pattern, and obtain the primary log pattern set in each time window.

12. The log pattern recognition system based on time window aggregation according to claim 11, characterized in that: The log pattern recognition system based on time window aggregation also includes: A primary counting module, used to obtain the number of occurrences of each primary log pattern in a single time window according to the primary log pattern set in each time window; The primary anomaly detection module is used to perform log anomaly detection based on the number of occurrences of each primary log pattern in a single time window; A historical pattern building module is used to build a historical log pattern list according to the primary log pattern; A similarity comparison module is used to compare the similarity between the primary log pattern and the historical log pattern of the log in the same time window, and merge the primary log pattern into the historical log pattern list according to the similarity; The counting module is used to obtain the number of occurrences of each log pattern in a single time window based on the historical log pattern list; The anomaly detection module is used to perform log anomaly detection based on the number of occurrences of each log pattern in a single time window.

Images