Log normalization system and log normalization method

Abstract

To improve accuracy and processing speed related to searching when searching for a pattern that matches a log.SOLUTION: A log normalization apparatus for normalizing a log output from an information processing system into a structured log comprises: a log input unit for receiving a log output from an information processing system; a storage unit for storing pattern information, which are patterns of a plurality of logs, and first priority information indicating a first priority for pattern matching corresponding to a log transmission source; a pattern matching unit for, when the received log is a log transmission source corresponding to the first priority information, performing pattern matching between the log and the patterns in the pattern information using a pattern corresponding to the first priority and converting the log into a normalized log using the matched pattern; and a log output unit for outputting the normalized log converted by the pattern matching unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for processing a log output from an information processing system. [Background technology]

[0002] Logs output from a system are used for system monitoring and debugging. For example, Non-Patent Document 1 discloses a method of searching for a matching template for an input log using TF-IDF (Term Frequency - Inverse Document Frequency), analyzing the log, and converting it into a structured log. Patent Document 1 also discloses a method including: importing data including logs acquired from multiple systems via a network by a data ingestion and query system; receiving user input indicating a range for acquiring data and criteria expressed in a structured language by the data ingestion and query system; acquiring data based on the range indicated by the user input; extracting a first field value including a first numerical value and a second field value including a first dimension indicating a measured characteristic of a computing device from the searched data based on the criteria and range indicated by the user input by the data ingestion and query system; and storing the first structured metric and the first dimension in a time series metric store by the data ingestion and query system, wherein the first structured metric includes a first numerical value and the first dimension is associated with the first numerical value. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent Application Publication No. US20190163678A1 (Generating structured metrics from log data) [Non-patent literature]

[0004] [Non-Patent Document 1] D. Schipper, M. Aniche and A. van Deursen, "Tracing Back Log Data to its Log Statement: From Research to Practice," 2019 IEEE / ACM 16th International Conference on Mining Software Repositories, 2019, pp. 545-549 Summary of the Invention [Problem to be solved by the invention]

[0005] When an information processing system uses many types of software or when a large number of information processing systems are monitored, many types of logs are output, and many types of patterns (templates) are required to normalize (structure) the output logs. In this regard, Non-Patent Document 1 discloses a method of searching for a template that matches an input log using TF-IDF, analyzing the log, and converting it into a structured log. Similarly, Patent Document 1 discloses a method of searching for a Source Type that matches an input log using an undisclosed method, and generating metadata equivalent to a structured log.

[0006] However, in both Non-Patent Document 1 and Patent Document 1, the time required to search for a pattern that matches the input log increases in proportion to the number of patterns (templates, source types), and the possibility of searching for an incorrect pattern also increases. For this reason, for example, when normalizing a log using data with preset huge patterns that include all log patterns that can be output by well-known tools, middleware, and libraries, the processing speed and accuracy of the search can become an issue.

[0007] An object of the present invention is to improve the accuracy and processing speed of a search when searching for a pattern that matches an input log from a huge number of patterns. [Means for solving the problem]

[0008] The log normalization system of the present invention is a log normalization device that normalizes logs output from an information processing system into structured logs, and is characterized in that it comprises a log input unit that receives logs output from the information processing system, a memory unit that stores pattern information that is a pattern of a plurality of logs and first priority information that indicates a first priority of a pattern match corresponding to a log source, a pattern matching unit that, when a received log is a log source corresponding to the first priority information, performs pattern matching between the log and a pattern of the pattern information using the pattern corresponding to the first priority and converts the log into a normalized log using the matched pattern, and a log output unit that outputs the normalized log converted by the pattern matching unit. Effect of the Invention

[0009] According to the present invention, it is possible to improve the accuracy and processing speed of searches when searching for patterns that match a log. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration according to an embodiment of the present invention. [Diagram 3] FIG. 13 is a diagram showing an example of an operation screen in the embodiment of the present invention. [Figure 4A] FIG. 2 is a diagram showing an example of the configuration of configuration information in an embodiment of the present invention. [Figure 4B] FIG. 2 is a diagram showing an example of the configuration of configuration information in an embodiment of the present invention. [Diagram 5] FIG. 11 is a diagram showing an example of pattern information in the embodiment of the present invention. [Figure 6] FIG. 4 is a diagram showing an example of software information according to an embodiment of the present invention. [Figure 7]FIG. 11 is a diagram showing an example of history information according to an embodiment of the present invention. [Figure 8] 10 is a flowchart illustrating an example of processing by a software identification unit in the embodiment of the present invention. [Figure 9] 11 is a flowchart showing an example of a pattern matching process in a pattern matching unit in the embodiment of the present invention. [Figure 10] 11 is a flowchart illustrating an example of a pattern matching process according to an embodiment of the present invention. [Figure 11] 11 is a flowchart illustrating an example of processing by a pattern generating unit in the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] An embodiment of the present invention will now be described in detail with reference to the drawings.

[0012] However, the present invention is not limited to the following examples, and includes various modifications and equivalent configurations within the spirit of the appended claims. For example, the above-mentioned examples have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the described configurations.

[0013] In this embodiment, each piece of information is described in the form of a "table" or "text data in JSON format", but this information does not necessarily have to be expressed in a table data structure, and may be expressed in a data structure such as a list, DB (DataBase), or queue, or in text data in a format such as Yaml or XML, or in other formats. Therefore, to indicate that it does not depend on the data structure, "table", "list", "DB", "queue", etc. may be simply referred to as "information". In addition, when describing the content of each piece of information, it is possible to use the expressions "identification information", "identifier", and "ID (IDentification)", which are mutually interchangeable.

[0014] In this embodiment, the process executed upon pressing a button on a GUI (Graphical User Interface) may be executed upon calling a corresponding API (Application Programming Interface).

[0015] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0016] In addition, each of the configurations, functions, processing means, etc. described above may be realized in hardware, for example by designing some or all of them as an integrated circuit, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0017] In addition, in the following explanation, a program such as "XXX function" may be used as the subject of the explanation, but since a program is executed by the processor 201 to perform a defined process using the main storage device 204 and the communication control device 202, the explanation may be made with the processor 201 as the subject. Furthermore, the process disclosed with the program as the subject may be a process performed by a programming device.

[0018] In addition, when obtaining data or calling program functions between different electronic computers, a remote procedure call may actually be performed using a communication protocol such as a Web API. EXAMPLES

[0019] Logs (messages) output from information processing systems are used to monitor and debug the systems. Log formats include unstructured logs output as text, and structured logs structured using JSON (JavaScript (registered trademark) Object Notation). Below, they are shown as Example 1 and Example 2, respectively.

[0020] (Example 1) Unstructured log [2022-12-01T10:51:17.33] ERROR: table sample does not exist (Example 2) Structured log { "timestamp": 1669891877330, "log_level": "error", "table": "sample", "message": "ERROR: table sample does not exist" }

[0021] Compared to unstructured logs, structured logs are more suitable for monitoring and analysis using machine processing such as aggregation. For this reason, unstructured logs are converted (normalized) into structured logs as a pre-processing step for analysis. This log normalization can be achieved, for example, by using the techniques described in the above-mentioned Non-Patent Document 1 and Patent Document 2. Note that information processing systems include various systems and devices that output such logs.

[0022] FIG. 1 illustrates an example of a system configuration according to a first embodiment.

[0023] The log normalization device 1 is realized by a hardware configuration like a computer shown as an example in FIG. 2. The log normalization device 1 includes a control unit 2, a pattern matching unit 3, a software identification unit 4, an input / output unit 5 having a GUI function, a pattern generation unit 6, pattern information 7, software information 8, and history information 9. The pattern matching unit 3 receives a raw log 14 from an external device such as an APP system 11 running on a host 10, and outputs a normalized log 15 to an external storage 12. The software identification unit 4 receives configuration information 16 input from an external user terminal 13 used by a user such as an administrator via the input / output unit 5. For example, data exchange with the outside may use a Web API, may be read from a recording medium such as a magnetic disk device, or may be input on the screen of a user terminal if the configuration information is small. In the following, a case where the log normalization device 1 exchanges with the host 10, the storage 12, and the user terminal 13 will be described as an example, but the present technology is not limited to these and can be used in various environments in which logs are output.

[0024] When a raw log 14 is input to the pattern matching unit 3, the pattern matching unit 3 analyzes and converts the log using the pattern information 7, and outputs a normalized log 15 to a storage 12 such as an external storage device or the cloud (described in detail in FIG. 9).

[0025] The software identification unit 4 uses the history information 9 and the configuration information 16 to identify the software used in the host 10 or other log generation source, and registers the identification result in the software information 8 (described in detail in FIG. 8).

[0026] The input / output unit 5 accepts confirmation, addition, deletion and update of data in the pattern information 7, software information 8 and history information 9 based on user operations from the user terminal 13 or the like using a GUI function (described in detail in FIG. 3).

[0027] The pattern information 7 is a table that holds patterns, templates for analyzing raw logs (described in detail in FIG. 5).

[0028] The software information 8 is a table that holds candidates for software that is assumed to be running on the host to be processed (described in detail in FIG. 6). The software information 8 is used by the pattern matching unit 3 to reduce the search range for matching.

[0029] The history information 9 is a table that holds the raw log 14 received by the pattern matching unit 3 and the data of the pattern applied for normalization (described in detail in FIG. 7). The history information 9 is used by the software identification unit 4 to identify software.

[0030] The raw log 14 is data including a character string of an unstructured log and host information. The host information is information for identifying the source of the log, such as a source IP address, a host name, a container name, a Kubernetes (registered trademark) Pod name, or a combination of these, and is expressed in a data format such as JSON.

[0031] The normalized log 15 is data obtained by normalizing the raw log 14 into a structured log by the log normalization device 1, and includes character strings of the structured log. The normalized log 15 may also include, as metadata, information about the software that generated the log (such as the software name, version, URL of the source code, and line number). The metadata can be used, for example, as reference information for debugging.

[0032] The configuration information 16 is data used for identifying software in the software identification unit 4, and includes, for example, a dependency management file and a configuration file of the software for which a log is generated (described in detail in FIGS. 4A and 4B).

[0033] The data in the raw log 14, normalized log 15, and configuration information 16 may take the form of a string, a file, encoded binary data, or a data stream.

[0034] Here, in order to make the pattern search more efficient, the pattern information 7 may have index data in addition to the table. In this embodiment, the index is called a total index. The structure of the index varies depending on the pattern search method, and may take the form of, for example, a table with a TF-IDF value added for each pattern, an ordered tree based on the pattern structure, or a table in which patterns are simply arranged in ascending order of the priority 65 of the pattern information 7. The ordered tree based on the pattern structure is a tree structure adjusted so that a more inclusive pattern becomes a parent. For example, when there are patterns A ".*%{WORD:user}.*", B ".*%{WORD:user} log in.*", C ".*%{WORD:user} log out.*" and D ".*admin %{WORD:user} log in.*", A is the parent, B and C are the children of A, and D is the child of B. In addition, the order of sibling nodes is arranged so that the pattern with the smaller priority 65 (higher priority) shown in FIG. 5 is searched first.

[0035] 2 is a block diagram showing an example of a hardware configuration of the log normalization device 1. This hardware configuration may be a physical computer, or may be operated in units of computers that are logically divided from a physical computer called a virtual server. Alternatively, it may be a task (also called a process or container) executed on one computer or a cluster of multiple computers.

[0036] The electronic computer 20 includes a processor 21 such as a CPU (Central Processing Unit), a communication control device 22, a communication interface 23, a main memory device 24, and an auxiliary memory device 25. The processor 21, the communication control device 22, the communication interface 23, the main memory device 24, and the auxiliary memory device 25 are interconnected by an internal bus 26.

[0037] The processor 21 is hardware that controls operation. The main memory device 24 is a memory device that holds various programs and data, and is, for example, a semiconductor memory.

[0038] The auxiliary storage device 25 is a storage device having a large storage capacity, such as a hard disk device or a solid state drive (SSD). The auxiliary storage device 205 holds executable files of various programs. The auxiliary storage device 25 is accessible from the processor 201.

[0039] The communication control device 22 is hardware having a function of controlling communication, and is used for data exchange between the outside and the log normalization device 1 in FIG.

[0040] 3 is an example of an input screen 30, which is a GUI screen provided by the GUI function of the input / output unit 5 of the log normalization device 1. By using this screen, a user can confirm, add, delete, and update data of the pattern information 7, the software information 8, and the history information 9.

[0041] The input screen 30 includes add buttons (37, 39) for adding new data to tables storing pattern information 7, software information 8, history information 9, and various other information not shown, and delete buttons (38, 40, 41) for deleting existing data.

[0042] The input / output unit 5 displays the data of the pattern information 7 in the pattern information input unit 31. The input / output unit 5 accepts an operation from an operator using an input device such as a mouse or a keyboard, and enables selection 36 of an arbitrary cell 35 or row 34 in the pattern information input unit 31. The input / output unit 5 can update the value of the selected cell 35 by accepting input of characters or numbers from a keyboard or the like.

[0043] Furthermore, the input / output unit 5 can delete the selected row 34 by accepting pressing of the Delete button 38 with the row 34 selected. When the input / output unit 5 accepts pressing of the Add button 37 from the operator, it creates a new row in the pattern information input unit 31. The input / output unit 5 can add a new record to the pattern information 7 by accepting input of data for the newly added row from the operator. However, if the ID of the added row is not a unique value, the input / output unit 5 displays an error and skips the registration process. Data added, deleted, or updated by the pattern information input unit 31 is reflected in the pattern information 113.

[0044] The input / output unit 5 displays the data of the software information 8 in the software information input unit 32. As with the pattern information input unit 31, the input / output unit 5 can add, delete, or update the data of the software information 8 by accepting operations such as an add button 39, a delete button 40, or cell editing for the software information input unit 32. Similarly, the input / output unit 5 displays the data of the history information 9 in the history information input unit 33, and can delete or update the data of the history information 9 by accepting operations such as a delete button 41 or cell editing.

[0045] 4A and 4B are examples of configuration information 16. The format of configuration information 16 and a method of generating software information 8 using configuration information 16 in software identification unit 4 will be described using FIGS.

[0046] The configuration information 16 is data used for identifying software in the software identification unit 4, and includes, for example, a dependency management file and a configuration file of the software for which a log is generated. In FIG. 4A, as an example of a configuration file, (a) an example of a manifest used in Kubernetes (registered trademark) of a container orchestrator is shown. Also, as an example of a dependency management file, (b) an example of package.json used in Node.js is shown as in FIG. 4B. In addition to the manifest and package.json, the configuration information 16 also includes, for example, docker-compose.yml, Dockerfile, go.mod, pom.xml, requirement.txt, build.gradle, and the like.

[0047] The configuration information 16 includes host information 50 and software information 51. The host information 50 includes information for identifying the computer or process that is the source of the log generation (IP address, host name, container name, pod name, etc.). The software information 51 includes information on software and libraries used by the target indicated in the host information 50.

[0048] In (a) Kubernetes manifest in FIG. 4A, metadata.name can be used as host information 50, and spec.containers[*].image can be used as software information 51. In (b) package.json in FIG. 4B, name and dependencies can be used as software information 51. However, since package.json does not include host information 50, when using package.json as configuration information 16, host information 50 is added in a format such as JSON as additional information.

[0049] The software identification unit 4 receives the configuration information 16 via the input / output unit 5. When the configuration information 16 is input, the software identification unit 4 identifies the file type indicating the format of the configuration information 16 from the file name and data configuration, and then extracts and processes the host information 50 and the software information 51 by a process predetermined for each file type, and registers them in the software information 8. For example, in the case of a Kubernetes manifest, the processing includes a process of changing the key name of the host information 401A from metadata.name to pod, and aligning the data format of the host information 401A with the host information of the raw log 120. In addition, the processing includes a process of extracting the container image name (postgresql-10-rhel7) from spec.container[*].image (registry.access.redhat.com / rhscl / postgresql-10-rhel7:1) and normalizing the software name and version information ({"name":"postgres","version":10}) using a dictionary provided in advance by the software identification unit 4.

[0050] When initially registered in the software information 8, the value of the software priority 72, which will be described later with reference to Fig. 6, is set to be blank. This is because the value of the software priority 72 is a value added by an operation on the input screen 30 by the operator.

[0051] 5 shows an example of the data structure of the pattern information 7. The pattern information 113 holds a template (pattern) for analyzing the raw log 14, and is used by the pattern matching unit 3.

[0052] The pattern information 7 includes values ​​of an ID 60 , a pattern 61 , software 62 , a version 63 , a source code provider 64 , and a priority 65 .

[0053] The ID 60 is an identifier for each record, and has a unique value within the log normalization device 1 .

[0054] The pattern 61 is a character string that is a template for analyzing a log, and is used to extract parameters from an unstructured log. The pattern 61 may be, for example, a regular expression or a Grok (registered trademark) pattern.

[0055] Software 62 indicates the name of the software that generates the log of pattern 61, and version 63 indicates the version of the software that generates the log of pattern 61. Version 63 may be an enumeration of multiple values ​​(e.g., 1.0.0, 1.0.2) or a range (e.g., 1.0.0-2.3.5).

[0056] The source code provider 64 is information on the source code in which the output process of the log is implemented in the software 62. The source code provider 64 is indicated by a file name or a URL if the source code is published in a repository on the Internet, and may also include line number information.

[0057] The priority 65 indicates the degree to which the pattern 61 is given priority over other patterns. The smaller the priority 65, the higher the priority. For example, the inverse of the number of stars acquired by the repository of the software in GitHub (registered trademark) can be used. The priority affects the search order in the pattern matching unit 3.

[0058] If there are multiple log patterns for one software, you can set a priority for each log pattern. In this example, the software postgres has patterns with ID1 and ID4, and the priority is specified to be the same.

[0059] 6 shows an example of the data structure of the software information 8. The software information 8 holds candidates for software running on each host, and is used to reduce the pattern search range in the pattern matching unit 3.

[0060] The software information 8 includes values ​​of a log source 70, software candidate 71, and in-software priority 72. The log source 70 is information indicating a computer or process that is a log source, and is composed of, for example, a source IP address, a host name, a container name, a Kubernetes Pod name, or a combination of these. The data format of the log source 70 is expressed in, for example, JSON format.

[0061] The software candidate 71 indicates information about software and libraries running on the log transmission source 70, and includes information about a list of the names of the software. Also, for each piece of software, information about the corresponding version may be included along with the name.

[0062] The software priority 72 is a value for customizing the priority 65 of the pattern information 7 individually for each log transmission source 70, and is composed of a set of zero or more IDs and priorities. The ID indicates the ID 60 of the pattern information 7, and the priority indicates a priority to be newly set for the ID. The software priority 72 can be set by the input / output unit 5 accepting an edit in the software information input unit 30 from the operator on the input screen 30. In addition, when the input / output unit 5 accepts an operation in the history information input unit 33 from the operator on the input screen 30 and corrects an error in the history information 9, a priority value smaller than the priority 65 of the pattern registered before the change may be set for the changed pattern. This makes it possible to prevent the recurrence of an error in the pattern search.

[0063] In this example, the software priority is specified as [{id:1, priority:-1}], which specifies that the priority of the pattern registered in ID1 of the pattern information is to be set to "-1." Therefore, for the software postgres, the pattern with ID4 is given priority over ID1.

[0064] 7 is an example of a data structure of the history information 9. The history information 9 holds information on logs normalized by the log normalization device 1, and is used for content confirmation on the input screen 30 as well as for software identification by the software identification unit 4.

[0065] The history information 9 includes values ​​of a timestamp 80, a host 81, a log 82, and an applied pattern 83. The timestamp 80 is information about the time when the log normalization device 1 normalized the target log, and is expressed, for example, as a character string in ISO8601 format or a Unix Epoch number.

[0066] The host 81 is information representing a computer or process that is the sender of a log expressed in the same specifications as the log sender 70 of the software information 8. The log 82 is a character string of the raw log 14 input to the log normalization device 1. The applied pattern 83 is the ID 60 of the pattern information 7 used to normalize the log 82.

[0067] The process in which the software identification unit 4 performs the identification process using the history information 9 will be described in detail with reference to FIG.

[0068] Fig. 8 shows an example of software identification processing using history information 9 in software identification unit 4. This processing identifies software running on a target host from the tendency of patterns applied in log analysis, and updates software information 8. Note that one or more processes in the processing shown in Fig. 8 may be deleted, or the order of the processes may be changed. This also applies to the processing processes in the following Figs. 9, 10, and 11.

[0069] The software identification unit 4 performs the processes of S91 to S94 for each host at a predetermined fixed timing. Therefore, although S91 to S94 indicate the processes for one specific host, the same applies to the other hosts.

[0070] The timing of the software identification process is based on the value received by the input / output unit 5 from the user terminal 13, and the control unit 2 controls when the results of the identification process are reflected in the matching process, allowing for flexible response to addition of hosts and application systems. Of course, instead of the control unit 2, other units may perform similar control.

[0071] First, the software identification unit 4 acquires records having a specific host 81 from the history information 9 (S91). A limit may be set on the records to be acquired. The limit may be, for example, a number (e.g., the most recent 1000 records), a time (e.g., the past 12 hours), or a combination of these. This makes it possible to prevent old records from lowering the accuracy of software identification.

[0072] Next, the software identification unit 4 counts the number of records for each combination of software and version (S92). Specifically, the software identification unit 4 acquires records having a pattern 61 corresponding to an ID 60 having the same value as the applied pattern 83 of the acquired record from the pattern information 7. In the aggregation of the acquired records, if there are records having the same software and version values ​​corresponding to the pattern 61 included in the acquired pattern information 7 and having an inclusion relationship between the versions, the software identification unit 4 aggregates the records. For example, a record group whose software is postgres and whose version is 17.0.0 may be added to a record group whose software is postgres and whose version is in the range of 1.3.8-18.0.0. This process makes it possible to count the number of combinations of software and version corresponding to the applied pattern 83 for each host 81.

[0073] Next, the software identification unit 4 calculates software candidates for the host from the tabulation results (S93). For example, the software identification unit 4 selects, as software candidates, combinations of software and versions that have a certain number of records or more (e.g., 30 or more) in the tabulation results of S92, or that rank above a certain number (e.g., third place or higher) when the tabulation numbers of records are sorted in descending order. There may be multiple software candidates. If there is no match, the software identification unit 4 determines that there is no software candidate. Through this process, software candidates corresponding to the pattern applied when normalizing the raw logs for each host can be obtained based on past performance.

[0074] Finally, the software identification unit 4 registers the calculated software candidates in the software information 8 (S94). Specifically, if the software information 8 contains a record in which the host and log transmission source 70 have the same value, the software identification unit 4 updates (adds or overwrites) the calculated software candidates in the software candidates 71 of that record. On the other hand, if there is no corresponding record, the software identification unit 4 registers the host and software candidates as a new record. At this time, the software identification unit 4 expresses the software candidates in a data format such as JSON format.

[0075] Through the above process, the software identification unit 4 calculates software candidates for the host based on the history information 9, and updates the software information 8.

[0076] 9 shows an example of the pattern matching process in the pattern matching unit 3. This process converts the input raw log 14 into a normalized log 15 using pattern matching.

[0077] The pattern matching unit 3 receives the raw log 14 from outside and starts a series of processes (S101). The pattern matching unit 3 receives the raw log 14, for example, through a Web API or a file. When multiple raw logs 14 are input consecutively as a file or a data stream, the pattern matching unit 3 breaks down the raw log 14 into individual units based on the number of characters or a specific separator (for example, a line feed symbol), and performs the following processes on each raw log 14.

[0078] Next, the pattern matching unit 3 extracts host information from the received raw log 14 (S102).

[0079] Next, the pattern matching unit 3 extracts a pattern to be searched based on information about the software running on the host that generated the raw log 14 (S103). However, this process (S103) and the subsequent index creation (S104) may be performed when the software information 8 is registered in S94 by the software identification unit 4 (for example, after S94) other than during pattern matching. It is more efficient to have these processes performed by the software identification unit 3, but in order to clearly illustrate the features of the invention, the first embodiment describes them as processes performed by the pattern matching unit 3.

[0080] Specifically, the pattern matching unit 3 acquires from the software information 8 a record whose log transmission source 70 matches the host information extracted in S102, and obtains the software candidate 71. Thereafter, the pattern matching unit 3 extracts from the pattern information 7 a record whose software 62 and version 63 match the value of the software candidate 71. As described in FIG. 8, the software information 8 is updated based on the history information 9. Therefore, by performing this process, it is possible to obtain the software 62 and version 63 corresponding to the software candidate 71 obtained from the latest software information 8 based on past performance. In S103, one or more patterns 61 whose software 62 and version 63 match are extracted, and a pattern corresponding to the uninputted log 14 is searched for from among them in the process of S105 described later.

[0081] However, a match of versions 63 also includes the case where the version value of the software candidate 71 is included in the range or enumeration of versions 63. Furthermore, when there is no perfectly matching software 62, the pattern matching unit 3 may determine, for example, the similarity between the character string of the software name in the software candidates and the software 62 using the Levenshtein distance, and determine that the software 62 with a distance equal to or less than a certain value is the matching software.

[0082] Next, the pattern matching unit 3 creates an index for efficiently searching the extracted pattern (S104). The index created in this process is called an individual index. The structure and creation method of the individual index are the same as those of the overall index shown in FIG.

[0083] Next, the pattern matching unit 3 searches for a pattern corresponding to the uninputted log 14 using the index (S105). In searching for a pattern, the pattern matching unit 3 first searches the individual index, and if there is no corresponding pattern, searches the overall index. The search method is based on the format of the index. If the index is configured with TF-IDF, the pattern matching unit 3 extracts a pattern whose difference from the TF-IDF value of the uninputted log 14 is equal to or less than a certain value, and evaluates the pattern that can express the uninputted log 14 in ascending order of priority 65. If the index is an ordered tree, the pattern matching unit 3 evaluates whether the pattern can express the uninputted log 14 by searching with priority given to the depth from the root, and searches for a node that has no child or sibling node or cannot express the uninputted log 14. In the case of a table sorted in ascending order of priority, the pattern matching unit 3 evaluates whether the uninputted log 14 can be expressed from the top. If a corresponding pattern cannot be found, the pattern matching unit 3 returns an error, and subsequent processing is skipped.

[0084] Next, the pattern matching unit 3 extracts parameters from the raw log 14 based on the regular expression or Grok pattern of the pattern found in S105, and converts the parameters into a data structure such as JSON (S106). At this time, the pattern matching unit 3 may add metadata to the JSON as additional parameters. The metadata includes, for example, the original text of the raw log 14, host information, processing time, the name of the software determined to be the log generating source, the version of the software, the source code of the log generating source, and the like.

[0085] Next, the pattern matching unit 3 outputs the log converted in S106 as a normalized log 15, and registers the result in the history information 9. The pattern matching unit 3 registers, in the history information 9, the processing time as a timestamp 80, the host information contained in the unsent log 14 as a host 81, the original text of the log as a log 82, and the ID of the pattern into which the log was processed as an applied pattern 83 (S107).

[0086] Through the above processing, the pattern matching unit 3 can convert the raw log 14 into a normalized log 15.

[0087] Next, the pattern matching process of S105 will be described with reference to Fig. 10. Fig. 10 is an example of a flowchart illustrating the pattern matching process in this embodiment.

[0088] In the following, the pattern matching unit 3 performs matching of the raw log to be pattern matched by using software information, which is information on the program running at the log sender. The software information 8 stores an in-software priority 72, and which pattern stored in the pattern information 7 is related to the program running at the log sender is specified by linking it to the ID 60 of the pattern information 7. In addition, the priority order among the linked patterns is specified as the in-software priority 72.

[0089] The pattern matching unit 3 performs pattern matching on linked patterns among the patterns 61 in the pattern information 7 extracted in S103 in the order specified by the intra-software priority 72 in the software information 8 (S111). This makes it possible to appropriately find the pattern to be applied to the raw log in a short time.

[0090] The pattern matching unit 3 determines whether a pattern has been found by matching using the software information 8 and the pattern information 7 (S112), and if it is determined that the pattern has been found (S112; Yes), it ends the process. On the other hand, if it is determined that the pattern has not been found (S112; No), it performs pattern matching with the priority of 65 of the pattern information 7 (S113). Next, the pattern matching unit 3 determines whether a pattern has been found by pattern matching using the pattern information 7 (S115), and if it is determined that the pattern has been found (S115; Yes), it ends the process. On the other hand, if it is determined that the pattern has not been found (S115; No), it outputs an error (S116).

[0091] In this embodiment, an example is described in which the order in which pattern matching is performed when creating an index is controlled, but the control unit 2 may also change the order in which files storing the priorities used in pattern matching are used. EXAMPLES

[0092] In the second embodiment, pattern information 7 is generated from source code information 17. This allows the user to generate log patterns from software source code without creating pattern information 7 for each log. The second embodiment will be described with reference to the system configuration diagram of FIG. 1. In addition to the first embodiment, a pattern generation unit 6 and source code information 17 are required, but other components are the same as those in FIG. 1 and therefore will not be described.

[0093] The pattern generating unit 6 performs a process of generating pattern information 7 based on source code information 17 .

[0094] The source code information 17 is data that combines source code and metadata of software or libraries. The source code is a character string that represents a software program, and may be all or part of the information of files included in a repository of the software such as Git (registered trademark). The metadata is information that indicates the name and priority of the software, and may include, for example, the software name, version, source of source code, priority, and other information. The metadata may also be generated from the source code by the pattern generation unit 6.

[0095] The source code information data includes formats of character strings, files, encoded binary data, and data streams, and is input from outside as, for example, a Web API or a file.

[0096] 11 shows an example of a pattern generation process in the pattern generation unit 6. The pattern generation unit 6 generates pattern information 7 from source code information 17.

[0097] The pattern generating unit 6 receives source code information 17 from the outside via the input / output unit 5 and starts a series of processes (S121).

[0098] When the pattern generation unit 6 receives source code information 17, it checks whether the software name, version, source code acquisition source, and priority are included in the metadata of the source code information 17. If there is data that does not include this information, the pattern generation unit 6 attempts to generate metadata from the source code.

[0099] The pattern generation unit 6 obtains the software name from the repository name of the source code, configuration files in the source code (package.json, go.mod, setup.py, etc.), or titles in the README file (e.g., HTML H1 tag). When obtaining the software name from a configuration file, the pattern generation unit 6 identifies the software name by a process predetermined for each configuration file. For example, when the configuration file is package.json, the pattern generation unit 6 sets the name parameter to the software name. At this time, the pattern generation unit 6 may perform preprocessing on the software name, such as converting English words to lowercase or replacing spaces with underscores.

[0100] The pattern generation unit 6 also obtains the version and source code acquisition source from the configuration file in the source code in the same way as the software name, using a process that is predetermined for each configuration file. For example, if the configuration file is package.json, the pattern generation unit 6 sets the version parameter as the version and the homepage parameter as the software acquisition source.

[0101] Furthermore, when the software source is a repository with an evaluation function (number of stars, Weekly Downloads, etc.) such as GitHub, GitLab (registered trademark), or NPM (registered trademark), the pattern generation unit 6 can access the URL of the software source, obtain an evaluation value of the source code in a manner determined for each software source, and set the priority. For example, when the software source is GitHub, the pattern generation unit 6 may set the inverse of the number of stars as the priority. The priority value may be normalized to a range for each type of repository.

[0102] If the software name is not included in the metadata and cannot be generated from the software, the pattern generation unit 6 returns an error and skips the subsequent processes. If the version 63 and source code acquisition source 64 cannot be acquired, the pattern generation unit 6 leaves these fields blank. If the priority 65 cannot be acquired, the pattern generation unit 6 sets the default value to 1.

[0103] Next, the pattern generation unit 6 extracts a log message from the source code (S123). A log message is data that represents a document to be output as a log, and is composed of a character string, a variable, a function, a class, an attribute, or a combination of these. The pattern generation unit 6 identifies the log message through processes such as syntax analysis and semantic analysis that are predetermined for each programming language, and obtains the data and line number. For example, in the case of JavaScript, the pattern generation unit 6 can obtain arguments of functions (such as the debug function) of a logging library such as the console.log function or Log4js as a log message.

[0104] Also, when a log message is implemented by variables, functions, classes, attributes, etc., the pattern generation unit 6 may, for example, use the abstract syntax tree of the source code to identify variables, functions, classes, and attributes linked to the function that outputs the log, and acquire the values ​​and arguments of the variables and attributes as the log message. Furthermore, when a fixed string of a log message to be output at one time is divided into multiple parts, the pattern generation unit 6 may combine them into one and acquire it as a log message. The fixed string is, for example, a string that can be statically determined only from the source code, and does not include dynamically determined data such as variables. When multiple log messages are acquired from the source code, the pattern generation unit 6 performs the following process for each log message.

[0105] Next, the pattern generation unit 6 extracts parameters from the log message and assigns the parameter name and data type (S124). The parameters are values ​​extracted from the log message in the structured log, such as user and source_ip. The pattern generation unit 6 extracts source code information 17, with dynamically determined variables or functions embedded in the log message as parameters. At this time, the pattern generation unit 6 may use a syntax extraction tree to identify the data of each extracted parameter, and if the parameter has multiple data (e.g., a source IP address and a source port number), follow the syntax extraction tree to divide the parameter into multiple parameters for each data.

[0106] The pattern generation unit 6 obtains the parameter name from the name of the variable or function. At this time, the pattern generation unit 6 may select an appropriate parameter name from among predetermined parameter name candidates according to the name of the variable or function or the meaning of the log message in order to normalize the parameter name. The pattern generation unit 6 may use a process such as comparing each parameter candidate with the meaning of the variable, function, or log message using a character vector or distance in a concept dictionary using Word2Vec, BERT, or the like, and selecting the parameter with the closest meaning from the parameter candidates.

[0107] The pattern generation unit 6 determines the data type of the parameter from the variable type of the variable and the data type of the return value of the function. In addition to primitive values ​​such as characters, numbers, booleans, and binary, the data type may be, for example, a data type defined as a Grok pattern. In the case of a programming language that does not have variable types or in the case of a variable having a uniquely defined data type, the pattern generation unit 6 may specify the associated data type using a syntax extraction tree and identify it as the data type of the parameter.

[0108] Finally, the pattern generation unit 6 generates a pattern from the extracted log message and parameters, and registers the pattern together with the metadata in the pattern information 7 (S125). The pattern generation unit 6 can generate the pattern by replacing the parameters in the log message with regular expressions, Grok patterns, or the like. At this time, the pattern generation unit 6 may apply post-processing to the generated pattern to improve matching accuracy. In the post-processing, for example, by adding the regular expression ".*" to the beginning of the pattern, even if a prefix has been added to the log message outside the software, it can be treated as being matchable.

[0109] For registration in the pattern information 7, the pattern generation unit 6 registers the pattern as pattern 61, and the software name, version, source code, and priority of the metadata as software 502, version 503, source code 504, and priority 505, respectively. Then, the log normalization device 1 assigns a unique ID 60 to generate a record of the pattern information 7.

[0110] Using the above processes and information, the log normalization device 1 can generate the pattern information 7 from the source code information 17. This has the effect of eliminating the need for the user to create the pattern information 7 for each log.

[0111] Thus, according to this embodiment, the log normalization device 1 that normalizes the log output from the information processing system into a structured log includes a log input unit (pattern matching unit 3) that receives the log output from the information processing system, a storage unit (auxiliary storage device 25) that stores pattern information 7 that is a pattern of a plurality of logs and first priority information (software priority 72 of software information 8) that indicates a first priority of the pattern match corresponding to the log transmission source, the pattern matching unit 3 that performs pattern matching between the log and the pattern of the pattern information 7 using the pattern corresponding to the first priority when the received log is a log transmission source corresponding to the first priority information, and converts the log into a normalized log using the matched pattern, and a log output unit (pattern matching unit 3) that outputs the normalized log converted by the pattern matching unit 3. This can improve the accuracy and processing speed of searching when searching for a pattern that matches the log.

[0112] Furthermore, the pattern of the pattern information includes a second priority (priority 65 of pattern information 7) indicating the priority of pattern matching, and the pattern matching unit performs pattern matching using the second priority when a match is not found in pattern matching based on the first priority. This makes it possible to perform pattern matching using the second priority even when a match is not found in pattern matching based on the first priority.

[0113] The first priority may be a priority determined based on the software running on the log sender. By defining it in this way, the priority of the patterns to be applied can be determined according to the type of software that outputs the log, or according to the version type of the same software.

[0114] Moreover, the first priority may include an instruction to change the second priority associated with the pattern information included in the pattern information. By defining it in this way, the priority 65 of the pattern information 7 can be individually customized.

[0115] The first priority may be a priority order determined based on an evaluation of the software running at the software source. By defining it in this way, it is possible to determine a priority order that reflects the evaluation of the source code in a manner determined for each software source.

[0116] The device also includes an input / output unit 5 that receives update information of the second priority and changes the second priority included in the pattern information using the received update information. This makes it possible to change the priority to a value intended by the user.

[0117] In addition, the source code of the software running on the log sender is received, a pattern is created based on parameters included in the source code, and the created pattern is registered in the above-mentioned pattern information. This eliminates the need for the user to create pattern information 7 for each log.

[0118] Also, the received source code is semantically analyzed, a log message is output from the positions and names of variables and functions contained in the source code, parameters are extracted from the output log message, and a pattern of pattern information 7 is generated based on the output log message and the extracted parameters. This makes it possible to use the information obtained by the semantic analysis to estimate the positions and roles of parameters contained in the log message to be output, create a pattern, or find the parameters contained in the pattern.

[0119] Although the present invention has been described in detail above with reference to the drawings, the present invention is not limited to the above-mentioned various examples, and various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0120] 1 Log Normalizer 3. Pattern Matching Section 4 Software Identification Unit 5 Input / output section 6 Pattern Generation Section 7 Pattern Information 8 Software Information 9. Historical Information 14 Raw Logs 15 Normalized Logs 16 Configuration Information 17 Source Code Information

Claims

1. A log normalization device that normalizes a log output from an information processing system into a structured log, a log input unit that receives a log output from an information processing system; a storage unit that stores pattern information that is a pattern of a plurality of logs and first priority information that indicates a first priority of a pattern match corresponding to a log transmission source; a pattern matching unit that, when a received log is from a log transmission source corresponding to first priority information, performs pattern matching between the log and a pattern of the pattern information using a pattern corresponding to the first priority, and converts the log into a normalized log using the matched pattern; a log output unit that outputs the normalized log converted by the pattern matching unit; A log normalization device comprising:

2. a second priority indicating a priority of a pattern match in the pattern information; the pattern matching unit performs pattern matching using a second priority when a match is not found in the pattern matching based on the first priority; 2. The log normalization apparatus of claim 1.

3. The first priority is a priority determined based on software running on a log sender.

3. The log normalizer of claim 2.

4. the first priority includes an instruction to change the second priority associated with the pattern information included in the pattern information; 3. The log normalizer of claim 2.

5. The first priority is a priority determined based on an evaluation of software running at a software acquisition source; 4. The log normalizer of claim 3.

6. an input / output unit that receives update information of the second priority and changes the second priority included in the pattern information by using the received update information; 3. The log normalizer of claim 2.

7. receiving a source code of software running on a log transmission source, creating a pattern based on parameters included in the source code, and registering the created pattern in the pattern information; 3. The log normalizer of claim 2.

8. performing a semantic analysis of the received source code, outputting a log message based on the positions and names of variables and functions included in the source code, extracting parameters from the output log message, and generating a pattern of pattern information based on the output log message and the extracted parameters; 8. The log normalizer of claim 7.

9. A log normalization method for normalizing a log output from an information processing system into a structured log, comprising: A log input unit receives a log output from the information processing system, A storage unit stores pattern information that is a pattern of a plurality of logs and first priority information that indicates a first priority of a pattern match corresponding to a log transmission source; a pattern matching unit, when the log received by the log input unit is a log source corresponding to first priority information, performs pattern matching between the log and a pattern of the pattern information using the pattern corresponding to the first priority, and converts the log into a normalized log using the matched pattern; A log normalization method in which a log output unit outputs a normalized log converted by a pattern matching unit.

10. a second priority indicating a priority of pattern matching is included in the pattern of the pattern information, and when a match is not found in the pattern matching based on the first priority, the pattern matching unit performs pattern matching using the second priority.

10. The method of claim 9, wherein the log normalization is

Images