Data processing device and data processing program

Abstract

A data processing device of an embodiment of the present invention includes a storage circuit and a processor. The storage circuit can store a first template. The processor executes an analysis process on log data that was input. In the analysis process, the processor separates the log data into tokens (unigrams) that include a unit and position information of the unit, calculates a similarity SIM on the basis of the number of tokens in common between the log data and the first template and the number of tokens of the first template, and updates the first template on the basis of the similarity SIM.

Description

[Technical Field]

[0001] The embodiments relate to a data processing device, a data processing method, and a data processing program. [Background technology]

[0002] A computer may record a log of events that have occurred. A log is data that records and accumulates information about events that have occurred in chronological order, and is used, for example, to identify the cause of an abnormality that has occurred. Examples of log data include system logs, error logs, and access logs, and various data formats may be used. Because log data is large-scale data, there is a demand for faster analysis of log data. By speeding up the analysis of log data, the overall processing, including subsequent processing, can be sped up.

[0003] Known techniques for analyzing text-format log data include methods of extracting templates from log data and clustering them. Non-Patent Document 1 discloses a method using longest common substring (LCS). Non-Patent Document 2 discloses a method that formulates log data analysis as an optimization problem and uses evolutionary computing. Non-Patent Document 3 discloses a method of extracting desired templates using a prefix tree. Non-Patent Document 4 discloses a method that uses a dynamic inverted index. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] M. Du and F. Li, "Spell: Streaming Parsing of System Event Logs," 2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, ​​Spain, 2016, pp. 859-864, doi: 10.1109 / ICDM.2016.0103. [Non-patent document 2] S. Messaoudi, A. Panichella, D. Bianculli, L. Briand and R. Sasnauskas, "A Search-Based Approach for Accurate Identification of Log Message Formats," 2018 IEEE / ACM 26th International Conference on Program Comprehension (ICPC), Gothenburg, Sweden, 2018, pp. 167-16710 [Non-patent document 3] P. He, J. Zhu, Z. Zheng and MR Lyu, "Drain: An Online Log Parsing Approach with Fixed Depth Tree," 2017 IEEE International Conference on Web Services (ICWS), Honolulu, HI, USA, 2017, pp. 33-40, doi: 10.1109 / ICWS.2017.13. [Non-patent document 4] S. Huang et al., "Paddy: An Event Log Parsing Approach using Dynamic Dictionary," NOMS 2020 - 2020 IEEE / IFIP Network Operations and Management Symposium, Budapest, Hungary, 2020, pp. 1-8, doi: 10.1109 / NOMS47738.2020.9110435. Summary of the Invention [Problem to be solved by the invention]

[0005] However, the methods of Non-Patent Documents 1 to 4 have a problem in that the analysis takes a long time because they perform a search for a template similar to the log data and a calculation of the similarity between the log data and the template in a stepwise manner. [Means for solving the problem]

[0006] A data processing device according to an embodiment includes a memory circuit and a processor. A plurality of templates and a plurality of inverted indexes associated with different log data lengths; The processor executes an analysis process of the input log data. In the analysis process, the processor analyzes the log data into a word and simply The data is divided into tokens containing the position information of the words. a search is made for a template including common tokens whose words and position information match, for each token of the log data, from among a plurality of inverted indexes whose log data length matches that of the log data; and a first template is selected from the searched templates, which has the largest number of common tokens with the log data; Between the log data and the first template The above together Pass and calculating a similarity based on the number of tokens in the first template and the number of tokens in the second template; The log data does not match the first template, and Similarity If the value of the first threshold is exceeded, only the tokens common to the log data and the first template are retained. Update the first template. [Effects of the Invention]

[0007] The data processing device according to the embodiment can improve the efficiency of analyzing log data. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing an example of use of a data processing device according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of a hardware configuration of a data processing device according to the embodiment. [Figure 3] FIG. 3 is a block diagram illustrating an example of the functional configuration of the data processing device according to the embodiment. [Figure 4] FIG. 4 is a block diagram showing an example of a detailed configuration of a storage unit included in the data processing device according to the embodiment. [Figure 5] FIG. 5 is a table showing an example of an inverted index stored in a storage unit included in the data processing device according to the embodiment. [Figure 6] FIG. 6 is a table showing an example of templates stored in the storage unit included in the data processing device according to the embodiment. [Figure 7] FIG. 7 is a flowchart showing an example of analysis processing of the data processing device according to the embodiment. [Figure 8] FIG. 8 is a schematic diagram showing a specific example of part of the log data analysis process performed by the data processing device according to the embodiment. [Figure 9] FIG. 9 is a schematic diagram showing a specific example of a method for calculating the similarity of log data by the data processing device according to the embodiment. [Figure 10] FIG. 10 is a schematic diagram showing an example of a template updating method performed by the data processing device according to the embodiment. [Figure 11] FIG. 11 is a table showing the calculation time for similarity calculation between the embodiment and the comparative example. [Figure 12] FIG. 12 is a graph showing the results of efficiency evaluation of the embodiment and the comparative example. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described with reference to the drawings. The embodiments illustrate devices and methods for embodying the technical ideas of the invention. The drawings are schematic or conceptual. In the following description, components having substantially the same functions and configurations are designated by the same reference numerals.

[0010] <Embodiment> The data processing device 10 according to the embodiment will be described below.

[0011] <1> composition FIG. 1 is a schematic diagram illustrating an example of use of a data processing device 10 according to an embodiment. As illustrated in FIG. 1, the data processing device 10 is a computer capable of analyzing input log data and outputting templates. The data processing device 10 can extract similar portions from the input log data and generate multiple templates. The "*" included in each template indicates a wildcard. That is, the "*" is treated as a parameter in the template. For example, in the log data illustrated in FIG. 1, "ae3," "vl22," "ac3," and "ac1" correspond to parameters. Note that the log data is input to the data processing device 10 via, for example, a wired or wireless network. The data processing device 10 may analyze log data stored in an internal or externally connected storage device.

[0012] <1-1> Hardware configuration of data processing device 10 2 is a block diagram showing an example of a hardware configuration of a data processing device 10 according to an embodiment. As shown in FIG. 2, the data processing device 10 includes, for example, a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a communication device 14, and a storage device 15.

[0013] The CPU 11 is an integrated circuit capable of executing various programs. The CPU 11 controls the overall operation of the data processing device 10. The ROM 12 is a non-volatile semiconductor memory. The ROM 12 stores programs and control data for controlling the data processing device 10. The RAM 13 is, for example, a volatile semiconductor memory. The RAM 13 is used as a work area for the CPU 11. The communication device 14 is a communication circuit configured to be connectable to a network. The data processing device 10 transfers log data received via the communication device 14 to the RAM 13 or the storage device 15, and may output analysis results of the log data to an external device via the communication device 14. The storage device 15 is a non-volatile storage device. The storage device 15 stores, for example, system software for the data processing device 10 and log data acquired via the network. The data processing device 10 may have other hardware configurations. A display, an input interface, a removable storage device, etc. may be connected to the data processing device 10.

[0014] <1-2> Functional configuration of the data processing device 10 3 is a block diagram showing an example of a functional configuration of the data processing device 10 according to the embodiment. As shown in FIG. 3, the data processing device 10 includes, for example, a storage unit 20, an input unit 21, a token generation unit 22, an analysis unit 23, a management unit 24, and an output unit 25.

[0015] The storage unit 20 stores the inverted index dictionary IID and the template list TL used in the log data analysis process by the data processing device 10. The inverted index dictionary IID includes multiple inverted indexes. The inverted index is, for example, a hash table in which a word is used as a key and a set of templates containing the word is used as a value.

[0016] The input unit 21 accepts input of data Din. The input unit 21 extracts log data in text format from the data Din and inputs the extracted log data to the token generation unit 22. When extracting the log data, the input unit 21 may perform preprocessing such as excluding specific words and phrases or replacing specific words and phrases with predetermined words and phrases.

[0017] The token generation unit 22 converts the log data input from the input unit 21 from text format data into token sequence data. Then, the token generation unit 22 inputs the converted log data to the analysis unit 23. In the data processing device 10, a token is a unigram with position information, and corresponds to a pair of a word and information on the appearance position of the word. In this specification, the log length L of log data is defined by the number of tokens included in the log data. For example, if log data includes six tokens, the log length of the log data is L=6.

[0018] The analysis unit 23 analyzes the log data input from the token generation unit 22 using the inverted index dictionary IID and the template list TL stored in the storage unit 20. Then, the analysis unit 23 inputs the analysis result of the log data to the management unit 24.

[0019] The management unit 24 appropriately updates the inverted index dictionary IID and the template list TL stored in the storage unit 20 based on the analysis result of the log data input from the analysis unit 23. The management unit 24 also extracts template candidates based on the analysis result of the log data and inputs the extraction result to the output unit 25.

[0020] The output unit 25 outputs data Dout as the extraction result of the template candidates input from the management unit 24. The output destination of the extraction result of the template candidates by the output unit 25 can be changed as appropriate according to the settings of the program executed by the CPU 11.

[0021] In the data processing device 10, the processing of the storage unit 20 is realized by, for example, the RAM 13. The processing of the input unit 21 is realized by, for example, the CPU 11, the RAM 13, and the communication device 14. The processing of the token generation unit 22 is realized by, for example, the CPU 11 and the RAM 13. The processing of the analysis unit 23 is realized by, for example, the CPU 11 and the RAM 13. The processing of the management unit 24 is realized by, for example, the CPU 11 and the RAM 13. The processing of the output unit 25 is realized by, for example, the CPU 11, the RAM 13, and the communication device 14. The functional configuration of the data processing device 10 is not limited to this, and may be classified in other ways.

[0022] <1-3> Configuration of the storage unit 20 4 is a block diagram showing an example of a detailed configuration of the storage unit 20 included in the data processing device 10 according to the embodiment. As shown in FIG. 4, the inverted index dictionary IID includes a plurality of inverted indexes II, and the template list TL includes a plurality of templates T.

[0023] The illustrated inverted indexes II[5], II[6], and II[7] are associated with log lengths L=5, L=6, and L=7, respectively. That is, the inverted index dictionary IID has an inverted index II for each log length L. Hereinafter, the inverted index II associated with the log length L will be referred to as the inverted index II[L].

[0024] The illustrated templates T[5], T[6], and T[7] are associated with template IDs "5", "6", and "7", respectively. In this way, the template list TL includes multiple templates associated with template IDs. Hereinafter, a template T associated with a certain template ID will be referred to as "T[ID]".

[0025] Note that the details of the inverted index dictionary IID and the template list TL shown in the figure are merely examples. The types and number of inverted indexes II included in the inverted index dictionary IID may vary depending on the types and number of log lengths L detected based on the analysis results of multiple log data input to the data processing device 10. Furthermore, the number of templates included in the template list TL and the association of each template with a template ID may vary depending on the template management method.

[0026] <1-4> Construction of inverted index II 5 is a table showing an example of an inverted index II[L] stored in the storage unit 20 included in the data processing device 10 according to the embodiment. (A) and (B) of FIG. 5 show inverted indexes II associated with log lengths L=5 and L=6, respectively. The inverted index II[L] uses a token as a key and a set of templates including the token as a value. In the following description, it is assumed that addresses [0], [1], ..., are assigned in the order of appearance in the token sequence data as parameters representing the position of a word in the log data.

[0027] As shown in (A) of FIG. 5, in this example, the inverted index II[5] records the token (0, down) and the token (3, vlan). The token (0, down) corresponds to the word "down" located at address [0] in the log data. The template IDs 1, 4, 6 associated with the token (0, down) indicate that the templates T[1], T[4], and T[6] each contain the token (0, down). Similarly, the token (3, vlan) corresponds to the word "down" located at address [0] in the log data. 3] This corresponds to the word "vlan" located at the position (3, vlan). The template ID=1,4 ​​associated with the token (3, vlan) indicates that the templates T[1] and T[4] each contain the token (3, vlan).

[0028] As shown in (B) of FIG. 5, in this example, the inverted index II[6] records a token (0, Interface) and a token (2, Changed). The token (0, Interface) corresponds to the word “Interface” located at address [0] in the log data. The template ID = 0, 3, 5 associated with the token (0, Interface) indicates that the templates T[0], T[3], and T[5] each contain the token (0, Interface). Similarly, the token (2, Changed) corresponds to the word “Changed” located at address [2] in the log data. The template ID = 0, 5 associated with the token (2, Changed) indicates that the templates T[0] and T[5] each contain the token (2, Changed).

[0029] As described above, each template T associated with the inverted index II[L] has the same log length L as the inverted index II[L]. That is, the template ID referenced differs between the inverted index II[5] and the inverted index II[6]. In this example, one template list TL stores templates T with various log lengths L. The data processing device 10 may create a template list TL for each log length L. In this case, the template ID may be the same among multiple template lists TL, and the data processing device 10 manages templates for each log length L, for example, by using a pair of the ID of the template list TL and the template ID. The template ID may also be a hash value of a template character string (e.g., “Interface * changed to *”). The method of managing template IDs is not limited to a specific method and may be any method as long as it can execute the analysis process described below.

[0030] <1-5> Template T configuration Fig. 6 is a table showing an example of a template T stored in the storage unit 20 included in the data processing device 10 according to the embodiment. Fig. 6 shows an example of a template associated with the token (0, Interface) in Fig. 5(B).

[0031] As shown in Figure 6, template T[0] with template ID = 0 is associated with the template "Interface * changed to *". Template T[3] with template ID = 3 is associated with the template "Interface * move to position *". Template T[5] with template ID = 5 is associated with the template "Interface * changed state by *". The log length of these templates is L=6 because they are associated with the inverted index II[6]. In addition, the word located at address [0] of these templates is "Interface" because they are templates T associated with the token (0, Interface). In addition, the word located at address [2] of templates with template ID = 0,5 is templates T associated with the token (2, Changed), so they are both "Changed".

[0032] <2> operation The data processing device 10 according to the embodiment creates an inverted index for each log length, searches the inverted index for the template with the largest number of common tokens, and calculates the similarity from the number of common tokens and the number of tokens in the template. If there is no template that matches the log data, the data processing device 10 extracts the template with the highest similarity and, based on the similarity, combines the log data with an existing template to update it, or adds it to the template list as a new template.

[0033] The log data analysis process in the data processing device 10 according to the embodiment will be described in detail below. In this specification, it is assumed that when a computer that records logs uses the same template, the length of the log data is the same, i.e., the number of words in the log data is the same. It is also assumed that in the analysis process, the data processing device 10 processes the input log data line by line, i.e., sentence by sentence. It is also assumed that the data processing device 10 does not perform, for example, combining templates or dividing templates.

[0034] <2-1> Analysis process flow 7 is a flowchart showing an example of the analysis process of the data processing device 10 according to the embodiment. The flow of the analysis process of the data processing device 10 according to the embodiment will be described below with reference to FIG.

[0035] The CPU 11 starts (starts) the analysis process in response to the input of the log data. In other words, the analysis process starts in response to the input of the log data processed by the input unit 21 to the token generation unit 22.

[0036] First, the CPU 11 divides the input log data into unigrams with location information (step S10). Specifically, the token generator 22 divides the text sentence of the input log data into words and adds location information (address) to each divided character. The unigrams with location information correspond to the tokens of the log data.

[0037] Next, the CPU 11 calculates the log length L of the log data (step S11). Specifically, the analysis unit 23 counts the number of tokens included in the log data and outputs the count result as the log length L.

[0038] Next, the CPU 11 checks whether or not it is “L in IID” (step S12). Specifically, the analysis unit 23 checks whether or not the inverted index dictionary IID includes an inverted index II[L] corresponding to the log length L calculated in step S11.

[0039] If the processing in step S12 is not "L in IID" (step S12, NO), that is, if the inverted index II[L] is not included in the inverted index dictionary IID, the CPU 11 adds a template T[y] to the template list TL (step S13). In other words, the management unit 24 registers a new template T[y] in the template list. "y" corresponds to a template ID that is not used in the current template list TL.

[0040] Next, the CPU 11 stores the tokens of the log data in the template T[y] (step S14). In other words, the management unit 24 saves the sequence data of the tokens of the log data in the newly registered template T[y].

[0041] Next, the CPU 11 updates the inverted index II[L] (step S15). Specifically, the management unit 24 updates the inverted index II[L] with each token of the newly registered template T[y], and registers the association between the template T[y] and the existing tokens in the inverted index II[L].

[0042] Next, CPU 11 outputs ID=y as the analysis result (step S16). That is, output unit 25 outputs template ID=y of new template T[y] as a template that matches the log data input to data processing device 10. When the process of step S16 is completed, CPU 11 ends the series of processes in FIG. 7 (END).

[0043] If the processing in step S12 is "L in IID" (YES in step S12), that is, if the inverted index II[L] is included in the inverted index dictionary IID, the CPU 11 searches for a template ID corresponding to a token of the log data using the inverted index II[L] (step S20). Specifically, the analysis unit 23 searches the inverted index II[L] for a token that matches the token of the log data. Then, the analysis unit 23 reads out the template ID associated with the matched token from the inverted index II[L].

[0044] Next, the CPU 11 counts the number of common tokens NCT for each template T associated with the log data (step S21). A common token is a token that is common between the log data and the template, taking into consideration the order. In other words, a common token is the same word that appears in the same position in both the log data and the template. The number of common tokens NCT corresponds to the number of template IDs that appear in the group of template IDs read for each token in the log data. For example, the analysis unit 23 counts a template ID associated with only one token as "number of common tokens NCT = 1," and a template ID associated with two tokens as "number of common tokens NCT = 2."

[0045] Next, the CPU 11 calculates the similarity SIM (step S22). Specifically, the analysis unit 23 first selects a template T[x] from the template list TL. "x" corresponds to the template ID with the largest number of common tokens NCT. The analysis unit 23 then calculates the similarity SIM using the following formula (1) that uses the number of common tokens NCT and the number of tokens NT of the template T[x] and the log length L of the log data. SIM = {(NCT + L) - NT} / L ... (1)

[0046] Next, the CPU 11 checks the similarity SIM (step S23). Specifically, the management unit 24 checks whether the similarity SIM is “α <SIM<1”、“ SIMCheck which condition of “= 1” and “SIM ≦ α” is satisfied. “α” is a threshold value used for determining the similarity SIM, and is preset by the user, for example. Note that “ SIM = 1” indicates that the log data and the template T[x] match. “ SIM The smaller the value of “

[0047] If it is confirmed in the process of step S23 that “α < SIM < 1” (step S23, α < SIM < 1), the CPU 11 combines and updates the template T[x] with the largest common token number NCT with the log data (step S24). In other words, the CPU 11 integrates the log data into the existing template T[x]. Specifically, the management unit 24 uses the words that do not match as parameters between the template T[x] with a high similarity SIM and the log data. That is, the management unit 24 changes the tokens in the template T[x] that do not match the log data to parameters (wildcards).

[0048] Next, the CPU 11 updates the inverted index II[L] (step S25). Specifically, the management unit 24 reflects the updated template T[x] in the inverted index II[L] to correct the inconsistency between the inverted index II[L] generated by the update of the template T[x] and the template [x]. For example, the management unit 24 deletes ID = x from the set of template IDs with the tokens changed to wildcards in the inverted index II[L] as keys.

[0049] Next, the CPU 11 outputs ID = x as the analysis result (step S26). That is, the output unit 25 outputs the template ID = x of the existing template T[x] integrated with the log data as the template that matches the log data input to the data processing device 10. When the process of step S26 is completed, the CPU 11 ends the series of processes in FIG. 7 (end).

[0050] In the process of step S23, SIM If it is confirmed that "=1" (step S23, SIM =1), the CPU 11 proceeds to the process of step S26. That is, the CPU 11 outputs ID=x as the analysis result (step S26), and ends the series of processes in FIG.

[0051] If it is confirmed in the processing of step S23 that "SIM≦α" holds (step S23, SIM≦α), the CPU 11 proceeds to the processing of step S13. That is, the CPU 11 adds a new template T[y] to the template list TL based on the log data (step S13), stores tokens of the log data in the template T[y] (step S14), updates the inverted index II[L] (step S15), outputs ID=y as the analysis result (step S16), and ends the series of processing shown in FIG. 7 (END).

[0052] <2-2> Specific examples of analysis processing A specific example of the analysis process of the data processing device 10 according to the embodiment will be described below.

[0053] (Steps S10, S11, S20 and S21) 8 is a schematic diagram showing a specific example of part of the log data analysis process by the data processing device 10 according to the embodiment, and shows the processes corresponding to steps S10, S11, S20, and S21. In this example, a case where document data "Interface ae3 changed state to down" is input as log data will be described. (A), (B), (C), and (D) in FIG. 8 respectively show the analysis result of the input log data, the referenced inverted index II[6], the search result of tokens in the log data, and the count result of the number of common tokens.

[0054] When the process of step S10 is executed on the input log data, the log data is divided into unigrams (tokens) with location information, as shown in (A) of Fig. 8. Specifically, the log data "Interface ae3 changed state to down" is divided into tokens (0, Interface), (1, ae3), (2, changed), (3, state), (4, to), and (5, down).

[0055] As the log data includes six tokens, the CPU 11 detects that the log length of the log data is L=6 (step S11). Therefore, the CPU 11 refers to the inverted index II[6] associated with the log length L=6, as shown in FIG. 8B. In this example, the contents of the inverted index II[6] shown in the figure are the same as those of the inverted index II[6] shown in FIG. 5B.

[0056] Then, the CPU 11 searches for a template ID corresponding to each token in the log data using the inverted index II[6] (step S20). As a result, in this example, the search results shown in FIG. 8C are obtained. For example, because the token (0, Interface) is included in the inverted index II[6], template IDs = 0, 3, 5 are read out as the search results for the token (0, Interface) in the log data. Also, because the token (1, ae3) is not included in the inverted index II[6], "none" (for example, Null data) is assigned as the search result for the token (1, ae3) in the log data.

[0057] Then, the CPU 11 counts the number of common tokens NCT based on the search results for the template IDs read out in the table shown in Fig. 8(C) (step S21). As a result, in this example, the number of common tokens NCT for each template ID is calculated as shown in Fig. 8(D). In this example, the number of common tokens for template ID=0 is NCT=4, and the number of common tokens for template ID=3 is NCT=2. Then, the CPU 11 refers to the table shown in Fig. 8(D) and extracts (selects) the template with the largest number of common tokens NCT (ID=0 in this example) as the template with the highest similarity SIM, and executes the subsequent processing.

[0058] (Steps S22 and S23) 9 is a schematic diagram showing a specific example of a method for calculating the similarity SIM of log data by the data processing device 10 according to the embodiment, and shows the process corresponding to steps S22 and S23. In this example, a case will be described in which the similarity SIM between the log data "(0, Interface) (1, ae3) (2, changed) (3, state) (4, to) (5, down)" and the template "(0, Interface) (1, *) (2, changed) (3, state) (4, to) (5, up)" is calculated. Note that elements in the template that include "*", such as (1, *), are parameters of the template and are not counted in the number of tokens NT.

[0059] As shown in FIG. 9, there are four common tokens between the log data and the template: (0, Interface), (2, changed), (3, state), and (4, to). That is, in this example, the number of common tokens NCT=4. Since the address [1] of the template is a parameter, (1, ae3) of the log data is treated as a parameter. Then, the CPU 11 calculates the similarity SIM as shown in the following formula (2) using the number of common tokens NCT=4, the log length L=6, and the number of tokens NT of the template=5 (step S22). SIM = {(NCT + L) - NT} / L = (4 + 6 - 5) / 6 = 5 / 6 …(2) Thus, in this example, the similarity SIM = 5 / 6. Then, the CPU 11 determines whether to update the template based on the calculated numerical value of the similarity SIM (step S23).

[0060] (step S24) FIG. 10 is a schematic diagram showing an example of a method for updating the template T by the data processing apparatus 10 according to the embodiment, and extracts and shows the process corresponding to step S24. In this example, the log data "(0, Interface)(1, ae3)(2, changed)(3, state)(4, to)(5, down)" and the template "(0, Interface)(1, *)(2, changed)(3, state)(4, to)(5, up)" are selected, and the case where "α < SIM < 1" will be described.

[0061] As shown in FIG. 10, when the address [1] which is a parameter is excluded, the tokens that are different between the log data and the template are (5, down) of the log data and (5, up) of the template. In contrast, the CPU 11 updates the template to a template that only retains the common tokens (step S24). Specifically, the CPU 11 retains the common tokens (0, Interface)(2, changed)(3, state)(4, to), and sets the addresses [1] and [5] as parameters (wildcards). As a result, in the updated template, the token (5, up) of the template before update is changed to the parameter (5, *).

[0062] <3>Effect The data processing device 10 according to the embodiment uses an inverted index II for each log length L to search for a template with a high similarity SIM to the log data, thereby enabling high-speed template search. Furthermore, analysis using positioned unigrams reduces the number of template candidates for which the similarity SIM is calculated. Therefore, the data processing device 10 according to the embodiment can simply and quickly calculate the similarity SIM between the template search results and the log data. In other words, the data processing device 10 according to the embodiment can improve the efficiency of log data analysis. Furthermore, the data processing device 10 according to the embodiment can speed up various processes, such as anomaly detection and compression, thereby realizing high-speed processing of the entire system that analyzes log data.

[0063] (Example) FIG. 11 is a table showing the calculation time of similarity between the embodiment and comparative examples (Non-Patent Documents 1 to 4), and indicates the complexity of similarity calculation after template candidates are extracted. "Longest Common Substring" and "Levenshtein Distance" are common methods for calculating similarity between strings. "Drain" and "Paddy" correspond to Non-Patent Documents 3 and 4, respectively. "L" corresponds to the log length L.

[0064] As shown in Figure 11, the calculation time when using the longest common substring is O(L 2 ) The calculation time when using the Levenshtein distance is O(L 2 ) The calculation time when Drain is used is O(L). The calculation time when Paddy is used is O(L). The calculation time when the data processing device 10 according to the embodiment is O(1). As such, the calculation time of the similarity in the data processing device 10 according to the embodiment is shorter than any of the methods in Non-Patent Documents 1 to 4. In other words, the data processing device 10 according to the embodiment can calculate the similarity faster than the comparative example.

[0065] 12 is a graph showing the efficiency evaluation results of the embodiment and a comparative example (Drain). (A) of FIG. 12 shows the relationship between the data size of the log data for which the analysis process is performed and the execution time [sec] of the analysis process. (B) of FIG. 12 shows the relationship between the data size of the log data for which the analysis process is performed and the memory usage [MB] during analysis.

[0066] 12A, in both the embodiment and Drain, the execution time of the analysis process increases as the data size of the log data increases. The execution time of the analysis process in the embodiment is about 1 / 8 shorter than the execution time of the analysis process in Drain for all evaluated data sizes. In other words, the data processing device 10 according to the embodiment can execute the analysis process faster than Drain.

[0067] As shown in Figure 12 (B), when the data size is 300 KB, the memory usage of both the embodiment and Drain is similar. However, in Drain, the memory usage increases as the data size of the log data increases, while in the embodiment, the memory usage remains constant even when the data size of the log data increases. Therefore, it can be seen that the analysis process of the embodiment is performed more efficiently than Drain.

[0068] <4> others In the embodiment, the data processing device 10 handles text-format log data without distinguishing between uppercase and lowercase letters, but this is not limited to this. The data processing device 10 may distinguish between uppercase and lowercase letters when generating tokens. In the embodiment, the data processing device 10 calculates the similarity of a template with the largest number of common tokens NCT, but this is not limited to the maximum and may be a predetermined value or more.

[0069] The flowcharts and data tables used in the description of the analysis process in the embodiments are merely examples. The flowchart shown in FIG. 7 may have the order of processes changed as far as possible, or other processes added, as long as the same results as those in the embodiments are obtained. In this specification, the data processing device 10 may be referred to as a "server" or a "processing server." The CPU 11 may be referred to as a "processor." The ROM 12, RAM 13, and storage device 15 may each be referred to as a "storage circuit."

[0070] The hardware configuration of the data processing device 10 described in the embodiment is merely an example. The CPU 11 included in the data processing device 10 may be another circuit. For example, the data processing device 10 may use an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like instead of the CPU 11. The analysis processing described in the embodiment may be realized by dedicated hardware. The analysis processing of the data processing device 10 may be a mixture of processing performed by software and processing performed by hardware, or may be only one of them.

[0071] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention. [Explanation of symbols]

[0072] 10...Data processing device 11...CPU 12...ROM 13...RAM 14...Communication equipment 15...Storage device 20...Storage section 21...Input section 22...Token generation unit 23…Analysis department 24…Management Department 25...Output section

Claims

1. a processor that executes an analysis process of the input log data; a storage circuit capable of storing a plurality of templates and a plurality of inverted indexes associated with different log data lengths; In the analysis process, the processor Dividing the log data into tokens each including a word and position information of the word; searching for a template including a common token having matching word and position information for each token of the log data from among the plurality of inverted indexes whose log data length matches that of the log data; selecting a first template from the retrieved templates that has the largest number of common tokens between the first template and the log data; calculating a similarity based on the number of common tokens between the log data and the first template and the number of tokens in the first template; If the log data and the first template do not match and the similarity exceeds a first threshold, update the first template by retaining only tokens that are common between the log data and the first template. Data processing device.

2. On the computer, Dividing input log data into tokens each including a word and position information of the word; searching for a template including a common token having matching word and position information for each token of the log data from among a plurality of inverted indexes associated with different log data lengths and having a log data length that matches that of the log data; selecting a first template from the retrieved templates that has the largest number of common tokens between the first template and the log data; calculating a similarity based on the number of common tokens between the log data and the first template and the number of tokens in the first template; updating the first template by retaining only tokens common between the log data and the first template when the log data and the first template do not match and the similarity exceeds a first threshold; A data processing program that executes the above.

Images