Log analysis method and log analysis device

Abstract

The present invention reduces template errors due to log parsing by causing a computer to execute: a parsing procedure for parsing, into partial character strings, a portion that can be parsed into a fixed portion and a variable portion which are included in a log, using as a delimiter a blank, a symbol with no alphabet letters or numbers, or a capital alphabet letter; a determination procedure for determining, for each of the partial character strings, whether the partial character string is a variable portion, on the basis of whether the partial character string includes a number in a prescribed mode, or whether the partial character string is present in a prescribed corpus; and a replacement procedure for replacing a partial character string determined to be a variable portion with a character string indicating the variable portion.

Description

Log analysis method and log analysis device

[0001] The present invention relates to a log analysis method and a log analysis device.

[0002] Logs, which are semi-structured texts output from systems, software, devices, etc., record information about their execution and are important data that allow for checking the status and behavior of those systems at runtime. Logs obtained from large-scale, complex systems, etc., are often voluminous and complicated, and research into automated log analysis is underway to improve the efficiency of log analysis (Non-Patent Document 1). Log analysis after acquisition from a system, etc., involves the processes of log parsing and log mining.

[0003] Figure 1 shows the flow of log analysis. Log parsing involves a process called parsing, which breaks down the semi-structured text of each line of a log into a structured part, an unstructured part, a template part that is fixed text within that, and a parameter part where variables and the like are assigned.

[0004] Fig. 2 is a diagram showing an example of log parsing. Fig. 2 shows an example of log parsing performed on a line L1 in an HDFS log (Non-Patent Document 2). Line L1 is decomposed into a structured part (structured part) and an unstructured part (unstructured part), and the unstructured part is further decomposed into a template part (temp) and a parameter part (param).

[0005] Parsing transforms the log into a form that can be used as input for subsequent log mining (for example, the form of table data).

[0006] Log mining uses logs that have been transformed by log parsing as input data, and searches them using machine learning, data mining, etc. to extract meaningful patterns and trends, outputting the status and behavior of the system that is the source of the log output. The results obtained from log analysis are used for managing, monitoring, and troubleshooting systems.

[0007] He, S., He, P., Chen, Z., Yang, T., Su, Y., and Lyu, M. R., "A Survey on Automated Log Analysis for Reliability Engineering"、arXiv:2009.07237 , 2020[online]、インターネット、<URL:https: / / github.com / logpai / loghub / tree / master / HDFS>M. Nagappan and M. A. Vouk、"Abstracting log lines to log event types for mining software system logs"、2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010), 2010, pp. 114-117Tang, L., Li, T., and Perng, C.、"LogSig: Generating system events from raw textual logs"、Proceedings of the 20th ACM international conference on Information and knowledge management, 2011, pp. 785-794A. Makanju, A. N. Zincir-Heywood and E. E. Milios、"A Lightweight Algorithm for Message Type Extraction in System Application Logs"、in IEEE Transactions on Knowledge and Data Engineering, vol. 24, no. 11, pp. 1921-1936, Nov. 2012[online]、インターネット、<URL:http: / / english-corpora.org / wiki / >

[0008] Log parsing applies various log parsing techniques to multiple logs, such as frequent pattern mining (Non-Patent Document 3), which searches for frequently occurring words, clustering (Non-Patent Document 4), which performs grouping, and iterative log segmentation (Non-Patent Document 5). Based on the commonality of the formats of the multiple logs, the unstructured portions of the logs are decomposed into template and parameter portions. However, depending on the content of the log text, a template error can occur, in which parameters cannot be extracted and a template containing variables, etc. is created, as shown in Figure 3. When a template error occurs, the accuracy of log mining decreases due to the inclusion of variables in the template, affecting tasks such as system management. Because template errors are likely to occur unless log parsing is performed manually, a process is needed to reduce template errors during log parsing.

[0009] The present invention has been made in view of the above points, and has an object to reduce template errors caused by log parsing.

[0010] To solve the above problem, a computer executes a decomposition procedure for decomposing parts of a log that can be decomposed into fixed parts and variable parts into substrings using spaces, symbols excluding alphabets and numbers, or uppercase alphabetic characters as separators; a determination procedure for determining whether each of the substrings is a variable part based on whether the substring contains numbers in a predetermined manner or whether the substring exists in a predetermined corpus; and a replacement procedure for replacing the substring determined to be a variable part with a string indicating the variable part.

[0011] It can reduce template errors caused by log parsing.

[0012] FIG. 1 is a diagram showing a flow of log analysis. FIG. 2 is a diagram showing an example of log parsing. FIG. 3 is a diagram showing an example of a template error. FIG. 4 is a diagram showing an example of a hardware configuration of the log analysis device 10 according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of a functional configuration of the log analysis device 10 according to an embodiment of the present invention. FIG. 6 is a diagram showing an example of template data. FIG. 7 is a diagram showing an example of log table data. FIG. 8 is a flowchart for explaining an example of a processing procedure executed by the log analysis device 10. FIG. 9 is a diagram showing an example of removing template errors from template data. FIG. 10 is a diagram showing an example of template duplication removal processing.

[0013] An embodiment of the present invention will now be described with reference to the accompanying drawings. Fig. 4 is a diagram showing an example of the hardware configuration of a log analysis device 10 according to an embodiment of the present invention. The log analysis device 10 shown in Fig. 4 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, all of which are interconnected via a bus B.

[0014] The program that realizes the processing in the log analysis device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0015] When an instruction to start a program is received, the memory device 103 reads the program from the auxiliary storage device 102 and stores it. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the log analysis device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0016] Fig. 5 is a diagram showing an example of the functional configuration of a log analysis device 10 according to an embodiment of the present invention. In Fig. 5, the log analysis device 10 has a template error removal unit 11. The template error removal unit 11 is realized by a process in which one or more programs installed in the log analysis device 10 are executed by the processor 104. The log analysis device 10 also uses a database 12. The database 12 can be realized using, for example, an auxiliary storage device 102 or a storage device connectable to the log analysis device 10 via a network.

[0017] The database 12 stores log table data and template data obtained by parsing a set of logs (logs that are semi-structured text output from a system, software, device, etc.).

[0018] Fig. 6 is a diagram showing an example of template data. In Fig. 6, one line corresponds to one piece of template data. Each piece of template data includes a template ID and a template. <*> in the template indicates a variable portion where a variable is entered, and the rest of the portion is a fixed portion.

[0019] Fig. 7 is a diagram showing an example of log table data, which shows the results of breaking down a log into a sequence of structured parts (TIME, LEVEL, COMPONENT in the example of Fig. 7), a sequence of unstructured parts, a template ID, and a template.

[0020] The template error elimination unit 11 eliminates template errors from the template data and log table data stored in the database 12 .

[0021] The following describes the processing procedure executed by the log analysis device 10. FIG. 8 is a flowchart illustrating an example of the processing procedure executed by the log analysis device 10.

[0022] In step S101, the template error elimination unit 11 executes removal of template errors from the template data.

[0023] Fig. 9 is a diagram showing an example of template error removal in template data. Fig. 9 shows processing performed on the template data whose template ID is "0003" in Fig. 6. The template error removal unit 11 performs the following processing on the template of each line of the template data (the portion of the log that can be decomposed into a fixed portion and a variable portion).

[0024] First, the template error elimination unit 11 generates a variable search template by inserting \ before symbols and before uppercase letters in the template of the template data (S101-1). Symbols here are characters other than alphabets and numbers (for example, "[", "]", ":", "\", "$", "_", etc.). Note that \ is not added before the character string <*> that represents a variable.

[0025] Next, the template error elimination unit 11 breaks down the variable search template into words and performs a variable search (S101-2). Here, a word refers to a character string separated by spaces or a substring separated by a symbol other than <*>. In other words, if the template already contains a character string <*> indicating a variable portion, that character string is not included in the words. The symbol here includes the ¥ character added in step S101-1. Therefore, here, the original template (before S101-1 is executed) is broken down into substrings separated by symbols other than spaces, letters, and numbers, or uppercase letters. In FIG. 9, each word is underlined to indicate the range of the word.

[0026] In the variable search, the template error elimination unit 11 determines whether a word is a variable, for example, as follows. Note that one or more of the following determination methods may be used: (a) A variable if all characters in the word are numbers; (b) A variable if there are i or more consecutive numbers in the word (i is determined arbitrarily, for example, 3); (c) A variable if the word contains at least one number; (d) A variable if j% or more of the characters in the word are numbers (j is determined arbitrarily, for example, 60); (e) A variable if the word starts with 0x (a hexadecimal prefix); (f) A variable if there is a portion in the word where the characters are arranged in the order number, alphabet, number; (g) A variable if the word does not exist in a predetermined corpus (a database 12 of words and phrases used in natural language processing, for example, (Non-Patent Document 6, etc.). The above determination methods (a) to (f) are basically based on whether a word contains numbers in a predetermined manner, and each determination method is distinguished depending on the manner in which numbers are included (how numbers are included).

[0027] Next, the template error elimination unit 11 updates the template by replacing words determined to be variables with <*> and deleting \ from the variable search template (S101-3). Note that when \ appears consecutively, such as \\, this indicates that \ was present in the original template, so the template error elimination unit 11 deletes only one \.

[0028] When the above process has been performed on the templates for all rows of the template data, the process proceeds to step S102.

[0029] In step S102, the template error elimination unit 11 executes a template duplication elimination process to prevent the same template from existing in the template data (to eliminate redundancy in the template data).

[0030] FIG. 10 is a diagram illustrating an example of the template overlap removal process.

[0031] The template error elimination unit 11 checks whether there are any duplicate templates in the template data. If there are duplicate templates (if there are completely identical templates), the template error elimination unit 11 extracts the duplicate template group (hereinafter referred to as the "duplicate template group") and their respective template IDs (S102-1). The template error elimination unit 11 replaces the template in the log table data, in which any of the extracted template IDs is used, with the same template as in the duplicate template group (S102-2). The template error elimination unit 11 also selects one template ID from the duplicate template group (S102-3), and replaces any unselected template IDs in the duplicate template group with the selected template ID in the log table data (S102-4). Finally, the template error elimination unit 11 deletes the rows of the unselected template IDs from the template data (S102-5).

[0032] As described above, this embodiment can reduce template errors that can occur when unstructured parts of a log are decomposed into template parts and parameter parts during log parsing. This makes it possible to create templates without including variables, thereby improving the accuracy of log mining.

[0033] In this embodiment, the template error elimination unit is an example of a decomposition unit, a determination unit, and a replacement unit.

[0034] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0035] 10 Log analysis device 11 Template error removal unit 12 Database 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device B Bus

Claims

1. A log analysis method characterized by being executed by a computer, which comprises the steps of: a decomposition step of decomposing portions of a log that can be decomposed into fixed portions and variable portions into substrings using spaces, symbols excluding alphabets and numbers, or uppercase alphabetic characters as separators; a determination step of determining whether each of the substrings is a variable portion based on whether the substring contains numbers in a predetermined manner or whether the substring exists in a predetermined corpus; and a replacement step of replacing the substring determined to be a variable portion with a string indicating the variable portion.

2. The log analysis method according to claim 1, characterized in that the determination step determines that the substring containing numbers in the predetermined manner or the substring present in the predetermined corpus is the variable part.

3. The log analysis method according to claim 1, wherein if a part of the log that can be decomposed into a fixed part and a variable part contains a character string indicating the variable part, the character string is not included in the substring.

4. A log analysis device comprising: a decomposition unit configured to decompose parts of a log that can be decomposed into fixed parts and variable parts into substrings using spaces, symbols excluding alphabets and numbers, or uppercase alphabetic characters as separators; a determination unit configured to determine whether each of the substrings is a variable part based on whether the substring contains numbers in a predetermined manner or whether the substring exists in a predetermined corpus; and a replacement unit configured to replace the substring determined to be a variable part with a string indicating the variable part.

Images