Analytical device and analytical method

Abstract

To reduce the difficulty in detecting concurrent patterns between data items different in granularity.SOLUTION: An analysis device comprises: data acquisition units that acquire a plurality of data items; individual analysis units that extract a plurality of concurrent patterns in the acquired data items, and that extract the number of occurrences of the concurrent patterns; an integration unit that creates an integration log by, on the basis of the extracted number of occurrences, integrating the extracted plurality of concurrent patterns; and an integration analysis unit that extracts, from the integrated log, the concurrent patterns.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an analytical device and an analytical method. [Background technology]

[0002] The log analysis system described in Patent Document 1 aims to shorten the time required to extract combinations of log messages output consecutively within a certain period of time when analyzing log messages output from an information processing system. To achieve this aim, the log analysis system generates a reference pattern for each combination of log messages that appear synchronously based on the log message appearance information, and integrates the reference patterns based on the results of comparing the appearance information of the log messages included in the reference patterns between the reference patterns. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2016-075915 Summary of the Invention [Problem to be solved by the invention]

[0004] However, since the format of the logs output and the time intervals at which the logs are output differ for each of the above-mentioned information processing systems, it has been difficult to detect concurrent occurrence patterns among multiple logs with different formats, etc. output from multiple information processing systems.

[0005] An object of the present disclosure is to provide an analysis device and an analysis method that reduce the difficulty of detecting co-occurrence patterns between data with different granularities. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the analysis device according to the present disclosure includes a data acquisition unit that acquires data from multiple devices; an individual analysis unit that extracts, for each device, multiple simultaneous occurrence patterns and the number of times the multiple simultaneous occurrence patterns occur in the acquired data, and creates an integrated log by integrating the extracted simultaneous occurrence patterns based on the extracted number of times they occur; an integration unit that combines the integrated logs between multiple devices; and an integrated analysis unit that extracts simultaneous occurrence patterns from the combined log. [Effects of the Invention]

[0007] The analysis device according to the present disclosure reduces the difficulty of detecting co-occurrence patterns between data of different granularities. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a functional block diagram of an analysis apparatus BS according to an embodiment. [Figure 2] FIG. 2 is a functional block diagram of an individual log analysis unit 2 according to the embodiment. [Figure 3] FIG. 2 is a functional block diagram of an integrated log analysis unit 4 according to the embodiment. [Figure 4] 2 shows a hardware configuration of an analysis apparatus BS according to an embodiment. [Figure 5] 10 is a flowchart showing the operation of the analysis apparatus BS of the embodiment. [Figure 6] 1 illustrates the extraction and integration of event patterns according to an embodiment. [Figure 7] 10 illustrates an integrated log (first half) according to an embodiment. [Figure 8] 10 shows an integrated log (second half) according to an embodiment. [Figure 9] 10 shows the first half of an integrated log for multiple devices according to an embodiment. [Figure 10] 10 shows the integrated log (second half) of a plurality of devices according to an embodiment. [Figure 11] 10 illustrates the deletion of an event according to an embodiment. [Figure 12] 1 shows the effects of the analysis device BS of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] An embodiment of an analysis device according to the present disclosure will be described.

[0010] In the following, to facilitate explanation and understanding, multiple names may be collectively referred to by a single reference symbol; for example, the reference symbol "1" may be used to collectively refer to "data acquisition unit 1A," "data acquisition unit 1B," etc. Embodiment. <Embodiment> <Functions of the embodiment> The analysis device BS of the embodiment extracts co-occurrence patterns from the same data. In the analysis device BS of the embodiment, the processing is not divided into a learning phase and an inference phase.

[0011] A "co-occurrence pattern" refers to a combination of data that causes the same phenomenon. Hereinafter, "co-occurrence pattern" and "event pattern" are synonymous and may be abbreviated as "pattern."

[0012] FIG. 1 is a functional block diagram of an analysis apparatus BS according to an embodiment.

[0013] As shown in FIG. 1, the analysis device BS includes data acquisition units 1A to 1D, individual log analysis units 2A to 2D, integrated log creation units 3A to 3D, integrated log analysis units 4A to 4D, a pattern display unit 5, and a pattern storage unit 6.

[0014] <Configuration of data acquisition unit 1> The data acquisition unit 1 acquires at least one piece of data (for example, log data, metrics data) that is the analysis target BT from the system that is the target of analysis. The data acquisition unit 1 passes the acquired data to the log individual analysis unit 2.

[0015] <Configuration of Individual Log Analysis Unit 2> FIG. 2 is a functional block diagram of the individual log analysis unit 2 according to the embodiment.

[0016] As shown in FIG. 2, the log individual analysis unit 2 includes a first pattern extraction unit 20, a first pattern integration unit 21, a log integration unit 22, and an event pattern definition unit 23.

[0017] The first pattern extraction unit 20 classifies data with overlapping index ranges acquired from the data acquisition unit 1 and the log integration unit 22 into rows and converts them into classification values. The first pattern extraction unit 20 extracts combinations of events that form simultaneous occurrence patterns by performing frequent pattern mining on classification values ​​that occur in close time periods.

[0018] The method for extracting events may be association analysis, which is one of the machine learning techniques, or may be a method for matching with an external file that holds event patterns.

[0019] The first pattern integration unit 21 integrates patterns based on the occurrence frequency of the events or event patterns extracted by the first pattern extraction unit 20.

[0020] As a method of integrating patterns, for example, a method can be used in which matching subsets of event patterns are extracted as integration candidates, the occurrence frequency of each event is calculated and aggregated using time series data acquired from the data acquisition unit 1 or the log integration unit 22, and the common parts of the event patterns are merged based on the results of the calculation and aggregation.

[0021] The log integration unit 22 integrates the extracted pattern sequences into a single log row and rearranges them in index order or chronological order. The log integration unit 22 names the extracted patterns to classify the logs, and compresses the pattern definitions and pattern classification results. The log integration unit 22 saves the extracted patterns in the event pattern definition unit.

[0022] The event pattern definition unit 23 stores pattern definitions and the like in any location, such as a memory or a database.

[0023] The log storage unit 24 stores a log consisting of a collection of events from which no pattern was extracted. The stored log may be used by the first pattern extraction unit 20 as an element for specifying the analysis range, or may be discarded before further analysis.

[0024] <Configuration of Integrated Log Creation Unit 3> The integrated log creator 3 integrates the series of integrated logs created by the individual log analyzer 2 into one log line.

[0025] <Configuration of Integrated Log Analysis Unit 4> FIG. 3 is a functional block diagram of the integrated log analysis unit 4 according to the embodiment.

[0026] As shown in FIG. 3, the integrated log analysis unit 4 includes an event extraction unit 40 and a second pattern integration unit 41.

[0027] The event extraction unit 40 extracts events by performing frequent pattern mining on the data acquired by the integrated log creation unit 3.

[0028] The event extraction method may be association analysis, which is one of the machine learning methods, or may be a comparison with an external file that holds event patterns.

[0029] The second pattern integration unit 41 deletes unnecessary events from the extracted event pattern.

[0030] To delete events, events with matching event pattern subsets are extracted as integration candidates, the temporal occurrence frequency of each event is calculated and tallied from the time-series data acquired from the integrated log creation unit 3, and event patterns with significantly different occurrence frequencies are deleted. This allows only appropriate patterns to be output.

[0031] <Configuration of pattern display unit 5 and pattern storage unit 6> The pattern display unit 5 displays the extracted event pattern, and the pattern storage unit 6 stores the extracted event pattern.

[0032] Hardware Configuration of the Embodiment FIG. 4 shows the hardware configuration of the analysis apparatus BS according to the embodiment.

[0033] To perform the above-described functions, the analytical device BS of the embodiment includes a processor PR, a memory ME, and a storage medium KI, as shown in FIG. 2, and may further include an input unit NY and an output unit SY as necessary.

[0034] The processor PR is the well-known core of a computer, operating hardware according to software. The memory ME is composed of, for example, a dynamic random access memory (DRAM) and a static random access memory (SRAM). The storage medium KI is composed of, for example, a hard disk drive (HDD), a solid state drive (SSD), and a read-only memory (ROM). The storage medium KI stores a program PRG. The program PRG is a set of instructions that define the content of the processing to be executed by the processor PR.

[0035] The input unit NY is composed of, for example, a camera, a microphone, a keyboard, a mouse, and a touch panel, while the output unit SY is composed of, for example, an LCD monitor, a printer, and a touch panel.

[0036] Regarding the relationship between the functions and hardware configuration of the analytical device BS, on the hardware, the processor PR executes the program PRG stored in the storage medium KI using the memory ME, and, as necessary, realizes the functions of each part from the data acquisition part 1 to the pattern display part 5 by controlling the operation of the input part NY and the output part SY.

[0037] <Operation of the embodiment> FIG. 5 is a flowchart showing the operation of the analyzer BS according to the embodiment.

[0038] Step ST1: The data acquisition unit 1 acquires data (for example, sequence data of character strings or sequence data of numerical values) as an analysis target BT from an information system that is an analysis target.

[0039] Step ST2: Frequent pattern mining is performed on the acquired data, for example, time-series data, to extract simultaneous occurrence patterns and the number of occurrences.

[0040] Step ST3: Event patterns with matching subsets are extracted as integration candidates, and common portions of the patterns are merged based on the extracted occurrence counts.

[0041] FIG. 6 illustrates the extraction and integration of event patterns according to an embodiment.

[0042] As shown in FIG. 6, the simultaneous event patterns and the number of occurrences of the events are extracted from the log output from device X (step ST2).

[0043] The extracted occurrence counts are tallied using a tree structure. Specifically, one path from the top node to the leaf node corresponds to one combination pattern, and the number of occurrences of an event at a leaf node also represents the number of occurrences of that event combination pattern.

[0044] 6, "phenomenon" and "event" are the same. For example, phenomenon A output in the log data and event A expressed as a leaf node of the tree are the same.

[0045] For example, the pattern of the occurrence count of (D, E) in the extracted event pattern (A, D, E) matches the pattern of the occurrence count of (D, E) in the event pattern (D, E, F). Therefore, the two event patterns are combined to create an event pattern (A, D, E, F) (step ST3).

[0046] Step ST4: Names are assigned to the extracted patterns to classify the logs, and an integrated log is created in which the pattern definitions and the pattern classification results are compressed.

[0047] The definition of the extracted pattern is stored in the event pattern definition unit 23. Events that are not extracted as patterns are stored in the log storage unit 24.

[0048] FIG. 7 shows the integrated log (first half) of the embodiment.

[0049] FIG. 8 shows the integrated log (second half) of the embodiment.

[0050] Eight event patterns, more specifically, event patterns (phenomenon A, phenomenon B, phenomenon C), (phenomenon A, phenomenon B), (phenomenon A), (phenomenon A, phenomenon D), (phenomenon A, phenomenon D, phenomenon E), (phenomenon A, phenomenon D, phenomenon E, phenomenon F), (phenomenon D), and (phenomenon D, phenomenon F), are extracted from the log data of device X (shown in FIG. 6 ). The extracted patterns are given names (e.g., pattern P1, pattern P2), and pattern definitions “pattern P1: (phenomenon A, phenomenon B, phenomenon C), pattern P2: (phenomenon A, phenomenon B), pattern P3: (phenomenon A), pattern P4: (phenomenon A, phenomenon D), pattern P5: (phenomenon A, phenomenon D, phenomenon E), pattern P6: (phenomenon A, phenomenon D, phenomenon E, phenomenon F), pattern P7: (phenomenon D), pattern P8: (phenomenon D, phenomenon F)” are stored in the event pattern definition unit 23.

[0051] The pattern classification results are "{2021 / 1 / 100:00:01, 2021 / 1 / 100:00:03} Pattern P1;{2021 / 1 / 100:00:04, 2021 / 1 / 100:00:05} Pattern P2;{2021 / 1 / 100:00:06, 2021 / 1 / 100:00:06} Pattern P3;{2021 / 1 / 100:00:07, 2021 / 1 / 100:00:08} Pattern P4;{2021 / 1 / 10 {2021 / 1 / 100:00:07, 2021 / 1 / 100:00:09}Pattern P5;{2021 / 1 / 100:00:07, 2021 / 1 / 100:00:10}Pattern P6;{2021 / 1 / 100:00:11, 2021 / 1 / 100:00:11}Pattern P7;{2021 / 1 / 100:00:13, 2021 / 1 / 100:00:14}Pattern P8" are defined as an integrated log and sorted by index order or chronological order.

[0052] The event "2021 / 1 / 100:00:15 phenomenon G" that was not extracted as an event pattern is stored in the log storage unit 24 that holds a log consisting of a collection of events from which patterns were not extracted.

[0053] Step ST5: The same process as in step ST2 is carried out on the created integrated log.

[0054] Step ST6: The processes of steps ST2 to ST5 are repeated until there are no unclassified events or until frequent patterns can no longer be extracted. The series of processes in steps ST2 to ST6 are independent for each log, and the process of any log may precede or run in parallel.

[0055] Step ST7: The integrated logs generated up to step ST6 are integrated in chronological order, that is, a combined log is generated.

[0056] If there is only one log to be analyzed, the processes after step ST7 may be omitted.

[0057] FIG. 9 shows the integrated log (first half) for multiple devices according to the embodiment.

[0058] FIG. 10 shows the integrated log (second half) for multiple devices according to the embodiment.

[0059] It is assumed that by repeating steps ST2 to ST6 for device Y, the integrated log "{2021 / 1 / 100:00:02, 2021 / 1 / 100:00:03} pattern Q1; {2021 / 1 / 100:00:03, 2021 / 1 / 100:00:03} pattern Q2" is output.

[0060] In step ST7, the integrated logs generated from the data of device X and device Y are merged, and the integrated log of device X and the integrated log of device Y are combined. This results in an integrated log sorted in index order and chronological order, i.e., a combined log "{2021 / 1 / 100:00:01, 2021 / 1 / 100:00:03} Pattern P1;{2021 / 1 / 100:00:02, 2021 / 1 / 100:00:03} Pattern Q1;{2021 / 1 / 100:00:03, 2021 / 1 / 100:00:03} Pattern Q2;{2021 / 1 / 100:00:04, 2021 / 1 / 100:00:05} Pattern P2;{2021 / 1 / 100:00:06, 2021 / 1 / 100:00:06}Pattern P3;{2021 / 1 / 100:00:07, 2021 / 1 / 100:00:08}Pattern P4;{2021 / 1 / 100:00:07, 2021 / 1 / 100:00:09}Pattern P5;{2021 / 1 / 100:00:07, 2021 / 1 / 100:00:10}Pattern P6;{2021 / 1 / 100:00:11, 2021 / 1 / 100:00:11}Pattern P7;{2021 / 1 / 100:00:13, 2021 / 1 / 100:00:14}Pattern P8" are defined.

[0061] Step ST8: Frequent pattern mining is performed on the generated integrated log, that is, the combined log, to extract simultaneous occurrence patterns and the temporal occurrence frequency of each event.

[0062] Here, matching subsets of event patterns are extracted as integration candidates, and based on the temporal occurrence frequency of each extracted event, event patterns with a low probability of being simultaneous phenomena are deleted, and frequent pattern mining at a higher level is performed to extract event patterns.

[0063] Step ST9: It is determined whether or not there is a deviation in the temporal occurrence frequency of the event patterns.

[0064] FIG. 11 illustrates the deletion of an event in an embodiment.

[0065] 11, if the extracted event patterns are (A, B, C) = (9 times / 10 seconds, 5 times / 10 seconds, 1 time / 10 seconds) and (α, ε) = (80 times / 100 seconds, 40 times / 100 seconds), the temporal occurrence frequencies of the event patterns can be expressed as (A, B, C) = 1 time / 10 seconds, (A, B) = 4 times / 10 seconds, (A) = 4 times / 10 seconds, (α, ε) = 4 times / 10 seconds, (α) = 4 times / 10 seconds. Patterns that have a low probability of co-occurring with event pattern α can be considered to have temporal occurrence frequencies that are significantly different from the occurrence pattern of event pattern α, and therefore event patterns (A, B, C) with a large deviation in temporal occurrence frequency are deleted.

[0066] The threshold for the temporal occurrence frequency for deleting an event may be set arbitrarily.

[0067] If there is no candidate for deleting the event pattern, the process proceeds to step ST10 without deleting the event.

[0068] Here, for example, (A), (A, B) and (α) are calculated as follows. Event A is called A, event B is called B, and event C is called C, and (A, B) is the unit of combination, and A and B are combinations.

[0069] (A) = Frequency of occurrence of A only (A, B) = Frequency of occurrence of only A and B (A, B, C) = Frequency of occurrence of only A, B, and C In this case,

[0070] (A) = Frequency of occurrence of A - Frequency of occurrence of A and B = 9 times / 10 seconds - 5 times / 10 seconds = 4 times / 10 seconds

[0071] (A, B) = Frequency of occurrence of A and B - (A, B, C) = 5 times / 10 seconds - 1 time / 10 seconds = 4 times / 10 seconds

[0072] (α) = Frequency of occurrence of α - Frequency of occurrence of α and ε = 80 times / 100 seconds - 40 times / 100 seconds = 40 times / 100 seconds = 4 times / 10 seconds

[0073] Step ST11: The extracted event pattern is displayed by the pattern display unit 5 or stored in the pattern storage unit 6.

[0074] Effects of the embodiment FIG. 12 shows the effect of the analyzer BS of the embodiment.

[0075] As shown in Fig. 12, in conventional pattern extraction, frequent pattern mining is used after integrating different data, i.e., the log data of device X and the log data of device Y. Therefore, there is a high possibility that only disconnected co-occurrence patterns will be extracted.

[0076] As described above and shown in FIG. 12, the analysis device BS of the embodiment classifies data separately for each series, i.e., for each device X and each device Y, and then integrates the data. The analysis device BS of the embodiment uses frequent pattern mining on the integrated data. This makes it possible to extract co-occurrence patterns, for example, between character data and numeric data, in other words, between data of different granularity.

[0077] The analysis device BS of the embodiment also repeats the processes of event extraction, pattern integration, and log integration based on conditions, eliminating the need for a person to specify the number of times to extract simultaneous patterns and preventing patterns from being overlooked in extraction.

[0078] Generally, as log sizes tend to increase, concurrent patterns tend to be extracted in a fragmented state. However, the analysis device BS of the embodiment can efficiently extract concurrent patterns and improve analysis accuracy by merging common parts of fragmented event patterns. Furthermore, by deleting patterns whose occurrence frequencies of event patterns differ, analysis time can be shortened, and concurrent patterns can be extracted more efficiently. Furthermore, analysis time can be shortened by compressing the log while performing analysis.

[0079] <Variation 1> When the use is limited (for example, when there is only one log to be analyzed), the analysis device BS may be configured without the integrated log creation unit 3 and the integrated log analysis unit 4.

[0080] <Variation 2> Unlike the embodiment that uses a general machine learning method (for example, association analysis) as a frequent pattern mining method, the event patterns may be extracted by, for example, comparing them with an external file that holds the event patterns.

[0081] <Variation 3> Regarding the occurrence frequencies described above, it is also possible to use the occurrence frequencies calculated by the first pattern extraction unit 20 as they are, and delete patterns with deviations in occurrence frequencies. Therefore, the processing in step ST8 may be omitted, and combination candidates may be listed. Furthermore, it is also possible to reduce the amount of calculation by performing the processing in step ST8 after eliminating combination candidates through the analyses in steps ST9 and ST10.

[0082] The above-described embodiments may be combined with each other without departing from the spirit of the present disclosure, and components in each embodiment may be deleted or modified, or other components may be added, as appropriate. [Explanation of symbols]

[0083] 1 Data acquisition unit, 2 Individual log analysis unit, 3 Integrated log creation unit, 4 Integrated log analysis unit, 5 Pattern display unit, 6 Pattern storage unit, 20 First pattern extraction unit, 21 First pattern integration unit, 22 Log integration unit, 23 Event pattern definition unit, 24 Log storage unit, 40 Event extraction unit, 41 Second pattern integration unit, BS Analysis device, BT Analysis target, KI Storage medium, ME Memory, NY Input unit, P Pattern, PR Processor, PRG Program, Q Pattern, SY Output unit, X Equipment, Y Equipment.

Claims

1. a data acquisition unit that acquires data from a plurality of devices; an individual analysis unit that extracts, for each of the devices, a plurality of simultaneous occurrence patterns and the number of times the simultaneous occurrence patterns occur in the acquired data, and creates an integrated log by integrating the extracted simultaneous occurrence patterns based on the extracted number of times the patterns occur; an integration unit that combines the integrated logs between the plurality of devices; an integrated analysis unit that extracts the co-occurrence patterns from the combined logs; An analytical device comprising:

2. the individual analysis unit has a first pattern extraction unit, a first pattern integration unit, and a log integration unit, and repeats processing by the first pattern extraction unit, processing by the first pattern integration unit, and processing by the log integration unit until there are no unclassified events or until frequent patterns can no longer be extracted. The analytical device of claim 1 .

3. the integration unit merges common similar parts among the extracted multiple co-occurrence patterns; The analytical device of claim 1 .

4. the individual analysis unit extracts the temporal occurrence frequency of the simultaneous occurrence pattern; the integrated analysis unit excludes simultaneous occurrence patterns that cannot occur simultaneously based on the extracted temporal occurrence frequencies; The analytical device of claim 1 .

5. the integration unit names and classifies the co-occurrence patterns, and compresses the definitions of the co-occurrence patterns and the results of the classification. The analytical device of claim 1 .

6. The individual analysis unit has a log storage unit that stores a log consisting of a set of events from which the simultaneous occurrence pattern was not extracted, The stored log is treated as a series of combination patterns with one event in the same manner as other combination patterns in subsequent processing, and is used to specify the range of analysis, or is discarded and then analyzed. The analytical device according to claim 5 .

7. The data acquisition unit acquires a plurality of pieces of data, an individual analysis unit extracting a plurality of simultaneous occurrence patterns from the plurality of acquired data and the number of occurrences of the simultaneous occurrence patterns; an integration unit that integrates the extracted simultaneous occurrence patterns based on the extracted occurrence counts to create an integrated log; an integrated analysis unit extracting the coincidence patterns from the integrated logs; Analysis method.

Images