Template generation method, template generation program, and template generation system

Abstract

To provide a template generation method that generates a general-purpose log template with high accuracy from source codes of various applications.SOLUTION: In a template generation method, code information that can be substituted into a parameter of an existing plate is extracted from a source code using an abstract syntax tree of the source code. Then, the abstract syntax tree is searched based on the code information and the existing template, and a location of the source code related to the existing template is identified on the basis of a result of match determination between the code information and the existing template. The parameter in the existing template is verified on the basis of on processing contents of the source code at the identified location, and a modified template is created by modifying the parameter in the existing template on the basis of on a verification result, and is output.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a template generation method, a template generation program, and a template generation system. [Background technology]

[0002] When using system logs for anomaly detection or data mining, it is desirable for the logs to be structured for ease of handling. For this reason, the unstructured logs output by the system are parsed into structured logs. Specifically, a data structure for parsing called a template is created in advance, and parameters are extracted from the unstructured logs by pattern matching them with the template, and the unstructured logs are then converted into structured logs based on the parameters.

[0003] Such templates are often created by analyzing unstructured logs using clustering or machine learning.

[0004] However, templates created using clustering or machine learning are merely estimates based on unstructured logs and may not reflect the intent of the developer of the application that outputs the logs. Furthermore, parameter label information is not always accurately estimated. Estimating label information is particularly difficult for logs that only list values, such as access logs.

[0005] Therefore, a method for generating templates by analyzing the source code of an application has been proposed, as disclosed in Non-Patent Document 1. By analyzing the source code, it is possible to obtain the log definition intended by the developer, and also to appropriately infer parameter label information from variable names, function names, etc. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] D. Schipper, M. Aniche and A. van Deursen, “Tracing Back Log Data to its Log Statement:From Research to Practice,” 2019 IEEE / ACM 16th International Conference on Mining Software Repositories, 2019, pp. 545-549. Summary of the Invention [Problem to be solved by the invention]

[0007] However, source code can be written in a variety of ways, which makes it difficult to generate general-purpose log templates with high accuracy from the source code of various applications, regardless of the target application, source code language, framework, etc.

[0008] The present invention has been made in consideration of the above circumstances, and has as its object to generate general-purpose log templates with high accuracy from the source code of various applications. [Means for solving the problem]

[0009] In one aspect of the present invention, there is provided a template generation method executed by a template generation system that generates a template for converting an unstructured log output from a program running on an information processing system into a structured log. The template generation system has a processor and a memory, and includes: a reception process executed by the processor that receives input of source code of the program and an existing template, the existing template including portions that have been variableized using parameters; an abstract syntax tree creation process that creates an abstract syntax tree of the source code; an extraction process that uses the abstract syntax tree to extract, from the source code, code information that can be substituted for the parameters; a search process that searches the abstract syntax tree based on the code information and the existing template; a determination process that determines whether the code information matches the existing template; an identification process that identifies a location of the source code related to the existing template based on a result of the determination of match; a modified template creation process that verifies the parameters in the existing template based on the processing content of the source code at the location and creates a modified template by modifying the parameters in the existing template based on the verification result; and an output process that outputs the modified template. [Effects of the Invention]

[0010] According to the present invention, it is possible to generate general-purpose log templates with high accuracy from the source code of various applications. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing the configuration of a template generation system according to an embodiment. [Figure 2] FIG. 1 is a diagram showing the hardware configuration of a computer that realizes a template generation system according to an embodiment. [Figure 3] FIG. 2 is a diagram showing an abstract syntax tree according to the embodiment. [Figure 4A] FIG. 4 is a diagram showing literal information in code information according to the embodiment. [Figure 4B] FIG. 4 is a diagram showing identifier information in code information according to the embodiment. [Figure 5] FIG. 4 is a diagram showing template information according to the embodiment. [Figure 6] FIG. 4 is a diagram showing policy information according to the embodiment. [Figure 7] 1 is a flowchart showing the overall processing of a template generation system according to an embodiment. [Figure 8] 10 is a flowchart showing a template generation process according to the embodiment. [Figure 9A] FIG. 10 is an explanatory diagram (part 1) of the related code information extraction process according to the embodiment. [Figure 9B] FIG. 10 is an explanatory diagram (part 2) of the related code information extraction process according to the embodiment. [Figure 10] FIG. 10 is an explanatory diagram of a related code information extraction process involved in log generation according to the embodiment. [Figure 11] FIG. 4 is an explanatory diagram of a correction template generation process according to the embodiment. [Figure 12] FIG. 2 is an explanatory diagram of a GUI according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the following description and drawings are examples for explaining the present invention, and appropriate omissions and simplifications have been made for clarity of explanation. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the solution of the invention. The present invention is not limited to the embodiments, and all application examples consistent with the concept of the present invention are included within the technical scope of the present invention. Those skilled in the art can make various additions and modifications to the present invention within the scope of the present invention. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be plural or singular.

[0013] In the following description, a "CPU (Central Processing Unit)" is an example of one or more processor devices. The at least one processor device is typically not limited to a CPU, but may be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may be a processor core.

[0014] At least one processor device may be a circuit that is a collection of gate arrays written in a hardware description language that performs some or all of the processing. The circuit is a processor device in the broad sense, such as a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), or an application-specific integrated circuit (ASIC).

[0015] In the following description, a program is executed by a CPU to realize a processing function called "XXX unit" and become the executing entity of the process. The processing function may be realized by one or more computer programs being executed by a processor, or may be realized by one or more hardware circuits (e.g., FPGA or ASIC), or may be realized by a combination of these.

[0016] When a function is realized by executing a program by a processor, the specified processing is performed using a storage device and / or an interface device, etc., so the function may be considered to be at least a part of the processor. Processing described using a functional unit as the subject may also be processing performed by a processor or a device having that processor.

[0017] The program may be installed from a program source. The program source may be, for example, a program distribution computer or a computer-readable recording medium (e.g., a non-transitory recording medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0018] In the following explanation, various information may be explained in table format. A "YYY table" may be called "YYY information." The data format of the information may be a format other than a table format (for example, CSV (Comma Separated Values) format). Furthermore, various information may be stored in a memory unit as a table, or may be embedded as logic in a program.

[0019] In addition, in the following description, when describing elements of the same type without distinguishing between them, common reference symbols will be used, and when describing elements of the same type with distinction between them, reference symbols will be used.

[0020] (Configuration of Overall System S According to the Embodiment) 1 is a diagram showing the configuration of an overall system S according to an embodiment. The overall system S includes a template generation system 1, a user terminal 20, a code repository 30, and a package registry 40, which are connected via a network N.

[0021] The code repository 30 is a storage area that holds the source code 21, and is GitHub (registered trademark) or the like. The package registry 40 is a storage area that holds libraries, and is Npm, PyPI, or the like.

[0022] The template generation system 1 has a function of creating a template for converting an unstructured log output from a program running on an information processing system (not shown) into a structured log.

[0023] Template generation system 1 includes input / output unit 2, preprocessing unit 3, extraction unit 4, search unit 5, labeling unit 6, code information 7, template information 8, policy information 9, abstract syntax tree information 10, and dictionary information 11. Input / output unit 2, preprocessing unit 3, extraction unit 4, search unit 5, and labeling unit 6 are realized by processor 201 (FIG. 2) executing predetermined programs. Code information 7, template information 8, policy information 9, abstract syntax tree information 10, and dictionary information 11 are various types of information stored in a storage area within auxiliary storage device 205 (FIG. 2).

[0024] The input / output unit 2 receives various data (source code 21, existing template 81, sample log 23, policy 24) via a GUI 700 (FIG. 12) or an API (Application Programming Interface), and outputs a modified template 83 (FIG. 5).

[0025] The preprocessing unit 3 obtains missing code and libraries from the code repository 30 and package registry 40 based on dependency information (package.json, requirements.txt, etc.) included in the source code 21. The preprocessing unit 3 also creates an abstract syntax tree (AST) 100 ( FIG. 3 ) and stores it in the abstract syntax tree information 10.

[0026] The extraction unit 4 extracts literals 104 (FIG. 3) from the abstract syntax tree 100. The extraction unit 4 also extracts identifiers 105 (FIG. 3) from the abstract syntax tree 100. The extraction unit 4 also uses an existing template 81 and a sample log 23 to extract estimated parameters 82 (FIG. 5) corresponding to the existing template 81.

[0027] The search unit 5 extracts combinations of code information (literal information 7A (FIG. 4A) and identifier information 7B (FIG. 4B)) that are substitution candidates that can be substituted into an existing template 81 and its estimated parameters 82. The search unit 5 then identifies appropriate code information from these combinations based on the connections in the abstract syntax tree 100. The search unit 5 then creates a modified template 83 (FIG. 5) based on the identified code information.

[0028] The labeling unit 6 uses the dictionary information 11 to create a label 7B3 of the identifier information 7B (FIG. 4B) and a "label" of the estimated parameter 82 of the template information 8. In the case of the source code 21, the label ("label") is obtained by using a machine learning model or natural language processing to identify words associated with the identifiers (variable identifier 105A, function identifier 105B (FIG. 3)). In the case of the template (existing template 81 (FIG. 5)), the label ("label") is obtained by using a machine learning model or natural language processing to identify words associated with the values ​​of the estimated parameters 82 (FIG. 5). For example, if the identifier is getUser, a label such as "user" is assigned, and if the identifier is current_time, a label such as "time" is assigned.

[0029] The code information 7 includes literal information 7A and identifier information 7B. Details of the literal information 7A and identifier information 7B will be described later with reference to Figures 4A and 4B, respectively. Details of the template information 8 will be described later with reference to Figure 5. Policy information 9 will be described later with reference to Figure 6.

[0030] The abstract syntax tree information 10 includes an abstract syntax tree 100 (FIG. 3). The dictionary information 11 includes a machine learning model and natural language processing for realizing the processing function of the labeling unit 6.

[0031] The user terminal 20 transmits the source code 21, the existing template 81, the sample log 23, and the policy 24 to the template generation system 1.

[0032] The source code 21 includes the source code of the application for which the modified template 83 (FIG. 5) is to be generated, and dependency management files (package.json, requirements.txt, etc.). The user terminal 20 compresses the source code 21 and transmits it to the template generation system 1.

[0033] The existing template 81 is a template generated externally using clustering, machine learning, or the like. The existing template 81 has drawbacks, such as the content being an estimate, making it unclear whether it conforms to the intentions of the application developer, or the lack of parameter label information. An example of an existing template 81 is "User <*> logged in," where the asterisk "*" represents a variable (parameter).

[0034] The sample log 23 is a log of an application corresponding to an existing template 81. The sample log 23 is used to create estimated parameters 82 (FIG. 5). Examples of the sample log 23 include "User john logged in," "User bob logged in," and "User allice logged in."

[0035] The policy 24 is information for updating the behavior of processing such as search, and one or more records of the policy information 9 (FIG. 6) are transmitted as the policy 24 from the user terminal 20 to the template generation system 1.

[0036] (Hardware configuration of the computer 200 according to the embodiment) 2 is a diagram showing the hardware configuration of a computer 200 that realizes the template generation system 1 according to the embodiment. The computer 200 includes a processor 201, a communication control device 202, an I / F 203, a main storage device 204 such as a memory, and an auxiliary storage device 205 such as a storage device.

[0037] The processor 201 works in cooperation with the main memory device 204 to read and execute programs and data from the auxiliary memory device 205, thereby realizing various processing functions of the template generation system 1. The communication control device 202 controls communication between the computer 200 and other computers via the network N and the I / F 203.

[0038] (Abstract syntax tree 100 according to the embodiment) 3 is a diagram showing an abstract syntax tree 100 according to an embodiment. The abstract syntax tree 100 is a labeled, directed tree that represents the structure of a program. The abstract syntax tree 100 is used for program execution and static analysis. The abstract syntax tree 100 includes nodes 101 and edges 102.

[0039] The types and connection forms (connections) of nodes 101 in abstract syntax tree 100 differ depending on the tool used to create abstract syntax tree 100. Figure 3 shows abstract syntax tree 100 created based on source code 21 using the ast module of Python (registered trademark, the same applies hereinafter).

[0040] Nodes 101 represent the processing of a target program. The nodes 101 include code information 103. The code information 103 includes literals 104, identifiers 105, operators, and assignments. Edges 102 represent the order of operations between nodes 101, the flow of control, scope, etc.

[0041] The literal 104 is constant information such as a character string, a numerical value, a list, or an object that is directly written in the source code 21 of the node 101 .

[0042] The identifier 105 is name information such as a variable name, a function name, a class name, a library name, and various data types of the node 101. The variable identifier 105A is an identifier of a variable. The function identifier 105B is an identifier of a function.

[0043] A call 106 represents an operation to access target data, such as a variable call, a function call, or a class call. Generally, a call 106 does not necessarily exist in the abstract syntax tree 100. If a call 106 does not exist in the abstract syntax tree 100, the call 106 is created by searching the graph of the abstract syntax tree 100 based on the identifier 105 and the scope of the edge 102, and determining the corresponding caller.

[0044] (Literal information 7A according to the embodiment) 4A is a diagram showing literal information 7A of the code information 7 according to the embodiment. The literal information 7A stores literals 104 in the source code 21. Each record in the literal information 7A corresponds to a literal 104 (FIG. 3) in the abstract syntax tree 100. The literal information 7A has columns of a name 7A1, a node 7A2, and source code 7A3.

[0045] Name 7A1 stores literal 104. If multiple literals 104 are concatenated in name 7A1 (for example, "User" + "%s"), this concatenated value is stored. Node 7A2 is a node in abstract syntax tree 100 corresponding to name 7A1, and stores a list of corresponding nodes. Source code 7A3 is the file name and line number of source code 21 that includes literal 104 of name 7A1.

[0046] By extracting from the source code 21 the literal 104 that matches the literal portion (part other than the asterisk) of the existing template 81 of the template information 8 (FIG. 5), the relevant part of the log in the source code 21 is estimated.

[0047] (Identifier information 7B according to the embodiment) 4B is a diagram showing identifier information 7B of the code information 7 according to the embodiment. The identifier information 7B stores identifiers 105 in the source code 21. Each record in the identifier information 7B corresponds to an identifier 105 (FIG. 3) in the abstract syntax tree 100. The identifier information 7B has columns for name 7B1, type 7B2, label 7B3, node 7B4, and source code 7B5.

[0048] Name 7B1 stores identifier 105. Type 7B2 indicates the variable type of variable identifier 105A or the type of the return value of function identifier 105B. Label 7B3 is parameter label information estimated from name 7B1 by labeling unit 6 (FIG. 1).

[0049] By extracting the variable identifier 105A that matches the "type" and "label" of the estimation parameters 82 of the template information 8 (FIG. 5), the relevant part of the log in the source code 21 is estimated.

[0050] (Template information 8 according to the embodiment) 5 is a diagram showing template information 8 according to the embodiment. The template information 8 has columns of an existing template 81, estimated parameters 82, modified templates 83, and nodes 84.

[0051] The existing template 81 is a template for converting an unstructured log into a structured log, and is an existing template before modification. The existing template 81 includes a part that is variable using a parameter.

[0052] The estimated parameters 82 store the "type" (variable type) and "label" of the parameters, which are the variable parts of the existing template 81 estimated from the sample log 23, in JSON format or the like. The "type" is estimated by the extraction unit 4 (FIG. 1). The "label" is estimated by the labeling unit 6 (FIG. 1).

[0053] The modified template 83 is a template obtained by modifying the existing template 81 by the search unit 5 (FIG. 1).

[0054] Node 84 is a node of the abstract syntax tree 100 used in the modified template 83. It is held for display in a GUI 700 (FIG. 10) described below.

[0055] (Policy information 9 according to the embodiment) 6 is a diagram showing policy information 9 according to the embodiment. The policy information 9 has columns for name 91, phase 92, and action 93.

[0056] The name 91 is information specified by the user to manage the policy. The phase 92 is information about the timing of the processing to implement the policy. The phases stored in the phase 92 include the following. Note that the following is just an example. “Preprocessing”: The preprocessing unit 3 (Figure 1) acquires libraries and creates the abstract syntax tree 100. "Extraction": The extraction process of code information and the type estimation process of estimation parameters by the extraction unit 4 (FIG. 1). "Search": Extraction of related code information and search of the abstract syntax tree 100 by the search unit 5 (FIG. 1). "Labeling": Labeling process of identifier information and estimated parameters by the labeling unit 6 (Fig. 1). "Templatize": A process for creating a template based on the search results by the search unit 5 (Figure 1) (used when forcibly changing the modified template 83).

[0057] Action 93 is a process to be executed in a corresponding phase as a policy. Action 93 may be in a form in which code describing the process is given as a policy as is, or in a form in which only enabling or disabling a predefined policy is specified.

[0058] The "policy" is executed to improve the accuracy of parameter labeling by optimizing the process to match the source code 21. It is also executed to perform complementary processing when parameters cannot be obtained. It is also executed to perform overwriting processing when an unintended correction template 83 is output. It is also executed to give the GUI 700 (FIG. 12) described below the role of "visualizing the behavior of searches according to policy editing."

[0059] Specifically, the "policy" can execute forced changes in the correction template 83. If parameters cannot be inferred, the label can be completed from the inferred parameters 82. It can also add rules such as "Python class fields access variables under the __init__ function."

[0060] You can also add rules such as "When the parent node is Assign, the right child node is determined to be identifier 105 and the left is determined to be its value." Labeling can also be performed using your own machine learning model. You can also add rules such as "In the parameters of modified template 83, use name 7B1 (Figure 4B) of identifier 105 instead of label 7B3 (Figure 4B)."

[0061] (Overall processing of template generation system 1 according to the embodiment) 7 is a flowchart showing the overall processing of template generation system 1 according to the embodiment. In the overall processing shown in Fig. 7, processing is performed according to the type of data each time template generation system 1 (Fig. 1) receives data.

[0062] First, in step S11, the input / output unit 2 accepts data input from the user via the GUI 700 (FIG. 12) or API. Next, in step S12, the input / output unit 2 determines the data type of the input data. The data type is determined, for example, based on information about the data type of the input data specified by the user operating the GUI 700 or the like. If the data type is an existing template 81 and a sample log 23, the input / output unit 2 proceeds to step S13. If the data type is source code 21, the input / output unit 2 proceeds to step S14. If the data type is policy 24, the input / output unit 2 proceeds to step S19.

[0063] In step S13, the extraction unit 4 estimates the "type" of the estimated parameters 82 in the existing template 81 from the input existing template 81 and sample log 23. The labeling unit 6 also estimates the "label" of the estimated parameters 82. At this time, if a policy exists for the extraction and labeling phases, that policy is executed. Thereafter, the labeling unit 6 stores the existing template 81 and the estimated parameters 82 in the template information 8. After step S13 is completed, the template generation system 1 proceeds to step S18.

[0064] In step S18, input / output unit 2 executes template generation processing to generate modified template 83 from code information 7, template information 8, and abstract syntax tree information 10. In step S18, modified template 83 is generated for existing template 81 for which the information necessary for generating modified template 83 is stored in all of code information 7, template information 8, and abstract syntax tree information 10 at the time of executing step S18. Details of the modified template generation processing will be described later with reference to FIG. 8. When step S18 ends, template generation system 1 terminates the overall processing of template generation system 1.

[0065] In step S14, if the source code 21 received in step S11 includes a dependency management file (configuration file) and the corresponding source code 21 is not input in the dependency management file, the preprocessing unit 3 (FIG. 1) executes the following: It acquires data such as the corresponding library and code from the code repository 30 or package registry 40 described in the dependency management file. The preprocessing unit 3 then adds the data acquired from the code repository 30 or package registry 40 to the source code 21 input in step S11. It acquires the library or source code that is missing when executing the source code 21 from the code repository 30 or package registry 40 and adds it to the source code 21. At this time, if a policy exists in the preprocessing phase, the preprocessing unit 3 executes the policy.

[0066] By obtaining the missing library or code from the code repository 30 or package registry 40 and adding it to the source code 21 in this way, the accuracy of the source code analysis can be improved, and ultimately the accuracy of the modification template 83 can be improved.

[0067] Next, in step S15, the preprocessing unit 3 creates an abstract syntax tree 100 from the source code 21 to which data such as libraries and code has been added in step S14. The abstract syntax tree 100 may be created using a function provided by a programming language (such as the ast module in Python), or a general-purpose AST conversion tool that is language agnostic, such as antlr4. At this time, if there is a policy for the preprocessing phase, the preprocessing unit 3 executes that policy.

[0068] Next, in step S16, extraction unit 4 (FIG. 1) extracts literal 104 from node 101 of abstract syntax tree 100 and stores it in literal information 7A of code information 7. Literal 104 may include character strings, numbers, logic, arrays (lists), objects (Dict, HashMap, etc.), and enumerations (Enum, etc.). At this time, if a policy exists in the extraction phase, that policy is executed.

[0069] Next, in step S17, extraction unit 4 extracts identifier 105 from node 101 of abstract syntax tree 100. Identifier 105 may include a variable name, function name, class name, or library name. Extraction unit 4 then obtains type 7B2 (FIG. 4B) of identifier 105 from the type information connected to node 101 corresponding to identifier 105. Labeling unit 6 also obtains label 7B3 (FIG. 4B) of this identifier 105. Extraction unit 4 then stores name 7B1 (FIG. 4B), type 7B2, and label 7B3 of the obtained identifier 105 in identifier information 7B. If a policy exists in the extraction phase at this time, that policy is executed. After completing step S17, template generation system 1 proceeds to step S18.

[0070] In step S19, the input / output unit 2 (FIG. 1) stores the policy received in step S11 in the policy information 9. The template generation system 1 executes action 93 in the processing phase described in phase 92 of each policy. After step S19 is completed, the template generation system 1 proceeds to step S18.

[0071] In step S13 and steps S14 to S17, action 93 is executed based on the policy already stored in policy information 9. The policy newly added to policy information 9 in step S19 is not retroactively reflected in the processing results of step S13 and steps S14 to S17 that have already been executed. However, this is not limiting, and the policy newly added to policy information 9 in step S19 may be retroactively reflected in the processing results of step S13 and steps S14 to S17 that have already been executed.

[0072] (Template generation process according to the embodiment (step S18)) 8 is a flowchart showing the template generation process (step S18) according to the embodiment. The template generation process processes the existing template 81 received in step S11 (FIG. 7) and the abstract syntax tree 100 (created in step S15 (FIG. 7)).

[0073] First, in step S181, the input / output unit 2 extracts, for each existing template 81, candidates for literal information 7A and identifier information 7B (related code information) (FIG. 4A) that can be substituted into the parameters of the template.

[0074] That is, in step S181, the input / output unit 2 extracts a literal 104 (FIGS. 3 and 4A) whose name 7A1 matches the literal portion Lp (a character string in which parameters are replaced with "*" (asterisk), see FIG. 9A) of the existing template 81. "Match" refers to "partial match or complete match." Also, "partial match or complete match" may simply be called "match." Also, "match" may include "similar."

[0075] The input / output unit 2 also extracts identifiers 105 (FIGS. 3 and 4B) whose type 7B2 and / or label 7B3 match the "type" and / or "label" of the estimated parameters 82 corresponding to the existing template 81. The input / output unit 2 may also extract records whose "type" matches only, or records whose "label" does not match exactly but whose labels are determined to have similar meanings using dictionary information, from the identifier information 7B. If a policy exists in the search phase, that policy is executed.

[0076] In this way, code information whose type 7B2 and / or label 7B3 match with the literal part Lp of the existing template 81 and the literal 104, and whose estimated parameters 82 corresponding to the existing template 81 match, is determined as a candidate for related code information. Therefore, it is possible to accurately extract code information 103 that is appropriate as a candidate for related code information.

[0077] The input / output unit 2 records the character positions of the name 7A1 of the literal 104 that matches the existing template 81.

[0078] The related code information extraction process (step S181) will be described with reference to Figures 9A and 9B. Figure 9A is an explanatory diagram of the related code information extraction process (step S181 (Figure 8)) according to the embodiment.

[0079] 9A, if the existing template 81 is "User<*>-san", the literal part Lp is "User*-san". In addition, in the abstract syntax tree 100 shown in FIG. 9A, the code information 103 includes four pieces of code information 103a, 103b, 103c, and 103d.

[0080] In the first step, the input / output unit 2 extracts the literal 104 whose name 7A1 matches the literal part Lp. In the example of Fig. 9A, the code information 103 extracted in this step is the code information 103b and 103c.

[0081] In the second stage, the input / output unit 2 extracts code information 103 whose type 7B2 and / or label 7B3 match the "type" and / or "label" of the estimated parameters 82 corresponding to the existing template 81. In the example of FIG. 9A, "name" in code information 103a matches the "type"="string" of the estimated parameters 82, and "-san" in code information 103b matches the "type"="string". Furthermore, "User" in code information 103c matches the "label"="user".

[0082] Therefore, as shown in FIG. 9B, the code information 103a "Assign "name"", the code information 103b "Const "-san"", and the code information 103c "Const "User"" are finally extracted as candidates for related code information.

[0083] Next, in step S182, the input / output unit 2 searches the edges 102 and calls 106 of the abstract syntax tree 100, and extracts related code information involved in log generation from the related code information candidates extracted in step S181.

[0084] Specifically, the input / output unit 2 performs a search process for each candidate for related code information extracted in step S181. Then, among the sets of mutually connected related code information (a state in which another candidate has been found at the search destination), the part or position of the source code 21 or abstract syntax tree 100 with the highest coverage rate of the existing template 81 is extracted as the related code information involved in log generation. The coverage rate refers to, for example, the proportion of parameters that can be used with the existing template 81. When the coverage rates are the same, for example, the set of related code information with the fewest number of nodes between each piece of related code information is extracted as the related code information involved in log generation.

[0085] The search identifies the location of source code 21 related to the existing template 81 based on related code information connected based on the calling relationships of variables, functions, or classes. That is, starting from each related code information, a process is performed to check sibling nodes (nodes with the same parent node) in the tree structure of the abstract syntax tree, as well as descendant nodes of the sibling node itself and its descendant nodes (child nodes and all further child nodes). In addition, the search may consider the following reference destinations (a) to (f) as connected child nodes, for example, but is not limited to these. (a) In the case of a function call, the function definition is referenced. (b) In the case of a function definition, it refers to the assignment destination of the function call arguments and return value. (c) In the case of a variable call, the variable definition is referenced. (d) In the case of a variable definition, refer to the variable invocation. (e) In the case of a class call, the field definition of the class is referenced. (f) In the case of a class definition, refer to the assignment destination of the instance.

[0086] In this way, by identifying the location of source code related to an existing template based on related code information that connects code information whose literal information and identifier information at least partially match based on a call relationship, the accuracy of identifying the location can be improved.

[0087] At this time, the input / output unit 2 treats the node 101 corresponding to the connected literal 104 or identifier 105 as being involved in log generation, even if it is not a candidate for related code information. If a policy exists in the search phase, the policy is executed every time a search is performed.

[0088] The related code information extraction process (step S182) involved in log generation will be described with reference to Fig. 10. Fig. 10 is an explanatory diagram of the related code information extraction process (step S182 (Fig. 8)) involved in log generation according to the embodiment.

[0089] 10, the code information 103c is set as the search starting point St. First, the sibling nodes "Add" and "Call "getName"" of the code information 103c "Const "User"" are checked.

[0090] Among the sibling nodes, the code information 103c2 "Call "getName"" of the destination is a "function call", so the code information 103c3 "Func "getName"" which is a function definition is referenced based on the call 106. This corresponds to the search (a) above.

[0091] Next, by searching for the descendant nodes of the destination code information 103c3 "Func "getName"", it is possible to confirm that the code information 103a and 103b are connected.

[0092] In the example of FIG. 10, code information 103c, 103a, and 103b are extracted as related code information (sometimes called "related node information") involved in log generation.

[0093] Next, in step S183, the input / output unit 2 assigns the related node information obtained as a result of the search in step S182 to the existing template 81, and outputs the modified template 83 in a format such as a Grok pattern. Note that the format is not limited to the Grok pattern. At this time, the input / output unit 2 may add, together with the modified template 83, source code 21 related to the system log of the modified template 83 to the modified template 83. If a policy exists in the templating phase, that policy is executed.

[0094] In step S183, the processing content of source code 21 is identified, and based on this processing content, errors or omissions of parameters in existing template 81 are verified. Then, based on the verification results, a modified template 83 is created by modifying the parameters in existing template 81.

[0095] The generation of the modified template 83 will be described with reference to Fig. 11. Fig. 11 is an explanatory diagram of the modified template generation process according to the embodiment. Before the modified template generation process is executed, if there is a corresponding policy, it is assumed that the policy has been applied.

[0096] First, for the parameter location ("<*>") of existing template 81, a search is made to see if a corresponding variable identifier 105A exists among the related node information candidates extracted in step S182. The "corresponding variable identifier 105A" is the variable identifier 105A whose position information is recorded in step S181. Then, as shown in FIG. 11(a), it is assumed that a corresponding variable identifier 105A ("_name (type: string, label: name)") has been found. Note that if a corresponding variable identifier 105A is not found, parameter assignment for that location is terminated.

[0097] Next, a Grok pattern corresponding to the "type" of the variable identifier 105A found in FIG. 11(a) is selected. The correspondence between "type" and Grok pattern is stored in advance. For example, the correspondence between "type" and Grok pattern is NUMBER for int and float, WORD for string, DATA for others, etc. In the example of FIG. 11(b), the pattern: WORD is selected.

[0098] Next, as shown in Fig. 11(c), the parameter part ("<*>") of the existing template 81 is replaced with the Grok format using the "label" of variable identifier 105A and the Grok pattern created in Fig. 11(b). Then, the parameter part ("<*>") replaced with the Grok format is registered in the template information 8 as a modified template 83, and then transmitted to the user terminal 20. Note that the parameter type and name (label) may be supplemented in the modified template 83.

[0099] (GUI 700 according to the embodiment) 12 is an explanatory diagram of a GUI 700 according to an embodiment. The GUI 700 includes a source code display section 710, a template display section 720, a policy display section 730, and a data transmission instruction section 740.

[0100] The source code display unit 710 displays the abstract syntax tree 100 of the source code 21. The source code display unit 710 distinguishes, for example, by shading out, related nodes 711A, which are nodes extracted as related code information in step S182 (FIG. 8) among the nodes 711 in the abstract syntax tree 100. If the node 711 in the abstract syntax tree 100 is a literal 104, a variable identifier 105A, or a function identifier 105B, the node may be labeled with a name 7A1 or 7B1. The related node 711A is a node used in generating the "Modified Template" of the selected line 751 indicated by the cursor 750. The contents of the "Modified Template" can also be directly rewritten by the user. In this case, the method of rewriting is added to the policy 73.

[0101] The template display section 720 displays the input existing template 81 ("Existing Template"), a modified template 83 ("Modified Template") obtained by modifying the existing template 81, and the identifier information ("Parameter source") that served as the basis for the parameters of the modified template 83. "Parameter source" lists records of identifier information 7B having node 7B4 included in node 84 of modified template 83, and outputs their labels and names in Key:Value format.

[0102] The policy display section 730 displays the contents of the policy information 9. When the policy information 9 is changed via the GUI 700, the "Modified Template" in the template display section 720 and the display contents in the source code display section 710 are updated in real time.

[0103] The data transmission instruction section 740 has radio buttons for specifying data such as an existing template 81 plus a sample log 23, source code 21, or policy 24. By specifying the data via the radio buttons, specifying a file path, and pressing the Upload button, the file at the specified file path is sent to the template generation system 1.

[0104] (Effects of the embodiment) In the above-described embodiment, correspondences between existing templates generated by machine learning or the like and code information in the abstract syntax tree obtained by source code analysis are found, and the existing templates are modified based on groups of corresponding code information. Therefore, by matching variable and constant information in the source code, it is possible to generate modified templates containing parameter information with high accuracy. Furthermore, it is possible to create general-purpose, highly accurate templates regardless of the type of software or library being targeted.

[0105] Although the embodiments of the present disclosure have been described above in detail, the present disclosure is not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the present disclosure. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present disclosure is not necessarily limited to those having all of the described configurations. Furthermore, some of the configurations of the above-described embodiments can be added to, deleted from, or replaced with other configurations.

[0106] Furthermore, the above-described configurations, functional units, processing units, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software by a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in memory, storage devices such as HDDs and SSDs, or recording media such as IC cards, SD cards, and DVDs.

[0107] In addition, in the above-mentioned drawings, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily show all the control lines and information lines that are actually implemented. For example, it may be considered that almost all components are actually connected to each other.

[0108] The above-described arrangement of the processing functions and data is merely an example, and the arrangement of the processing functions and data can be changed to an optimal arrangement in terms of the performance of the hardware and software, processing efficiency, communication efficiency, etc. [Explanation of symbols]

[0109] S: Overall system, 1: Template generation system, 2: Input / output unit, 3: Preprocessing unit, 4: Extraction unit, 5: Search unit, 6: Labeling unit, 7: Code information, 7A: Literal information, 7B: Identifier information, 8: Template information, 9: Policy information, 10: Abstract syntax tree information, 21: Source code, 24: Policy, 30: Code repository, 40: Package registry, 73: Policy, 81: Existing template, 82: Estimated parameters, 83: Modified template, 100: Abstract syntax tree, 103: Code information, 104: Literal, 105: Identifier, 105A: Variable identifier, 105B: Function identifier, 200: Computer, 201: Processor, 204: Main memory, 700: GUI.

Claims

1. A template generation method executed by a template generation system that creates a template for converting an unstructured log output from a program running on an information processing system into a structured log, comprising: the template generation system includes a processor and a memory; Executed by the processor, a receiving process for receiving input of the source code of the program and an existing template that is an existing template including a portion that has been changed to a variable using parameters; an abstract syntax tree creation process for creating an abstract syntax tree of the source code; an extraction process of extracting code information that can be substituted into the parameters from the source code using the abstract syntax tree; a search process for searching the abstract syntax tree based on the code information and the existing template; a determination process for determining whether the code information matches the existing template; an identification process for identifying a location of the source code related to the existing template based on the result of the match determination; a modified template creation process for verifying the parameters in the existing template based on the processing content of the source code at the location, and creating a modified template by modifying the parameters in the existing template based on the verification result; an output process for outputting the modified template; A template generation method comprising:

2. 2. The template generation method of claim 1, The processor: The code information uses literal information and identifier information in the source code. A template generation method comprising:

3. 3. The template generation method according to claim 2, further comprising: the processor: In the determination process, determining whether the code information matches or partially matches the literal portion of the existing template and the parameter types and labels; In the specific processing, The code information determined in the determination process to match or partially match the literal portion, the type, and the label is identified based on related code information connected based on a calling relationship of a variable, a function, or a class, and the location of the source code related to the existing template is identified. A template generation method comprising:

4. 2. The template generation method of claim 1, the processor: In the modification template creation process, Complete the parameter names and types to create the modified template. A template generation method comprising:

5. 2. The template generation method of claim 1, the processor: A library or code that is missing when executing the source code is obtained from a code repository or package registry and added to the source code. A template generation method comprising the steps of:

6. 2. The template generation method of claim 1, the processor: The source code related to the existing template and the modified template is displayed on a display unit. A template generation method comprising the steps of:

7. 2. The template generation method of claim 1, the processor: Executes a policy for optimizing the processes in each processing phase including the abstract syntax tree creation process, the extraction process, the search process, the determination process, the identification process, and the modified template creation process, or the processing results of the processes. A template generation method comprising the steps of:

8. A template generation program for causing a computer to execute the template generation method according to any one of claims 1 to 7.

9. A template generation system that creates a template for converting an unstructured log output from a program running on an information processing system into a structured log, the template generation system includes a processor and a memory; The processor: executes a receiving process for receiving input of the source code of the program and an existing template, which is an existing template including a portion that has been changed to a variable using parameters; Execute an abstract syntax tree creation process to create an abstract syntax tree of the source code; executing an extraction process that uses the abstract syntax tree to extract code information that can be substituted for the parameters from the source code; executing a search process for searching the abstract syntax tree based on the code information and the existing template; execute a determination process to determine whether the code information matches the existing template; performing an identification process for identifying a location of the source code related to the existing template based on the result of the match determination; execute a modified template creation process to verify the parameters in the existing template based on the processing content of the source code at the location, and create a modified template by modifying the parameters in the existing template based on the verification result; Execute an output process to output the modified template A template generation system comprising:

Images