Log field name generation method, system and electronic device based on knowledge graph

Abstract

The present invention provides a method, system and electronic device for generating log field names based on a knowledge graph. The method includes: extracting attribute information and label information from sample log data to establish an entity-relationship-attribute ternary array of the sample log data; fusing the ternary arrays of multiple sample log data to obtain a knowledge graph of the log data; obtaining log data to be named; extracting attribute information from the log data to be named, weighted matching the attribute information of the log data to be named with the knowledge graph of the log data to obtain a matching result; and generating the names of each field of the log data to be named according to the matching result. By processing the sample log data to obtain a knowledge graph of the log data, the fields can be named more accurately based on the knowledge graph, and more information can be obtained through the name of the field and the knowledge graph corresponding to the name in subsequent processing, which is more conducive to data aggregation, anomaly detection, root cause analysis, trend prediction, etc.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and specifically to a log field name generation method based on a knowledge graph, a log field name generation system based on a knowledge graph, and an electronic device. Background Art

[0002] With the rapid development of information technology and the Internet, various software systems are constantly being generated and integrated, and new challenges have been brought about in terms of failure frequency, system complexity, and big data environment. Operation and maintenance work has roughly gone through several stages: manual, tool and automation, platform-based, and intelligent operation and maintenance.

[0003] Logs are a very important and special type of operation and maintenance data source. Due to their wide range of sources, rich content, and flexible format, logs can play a unique role in data aggregation, anomaly detection, root cause analysis, trend prediction, and other aspects of intelligent operation and maintenance. Parsing logs is the basis for the above work, including log structuring, log template extraction, etc., and the naming of log fields is an important part of it.

[0004] In the existing technical solutions, usually only the format information of the log field itself is used for abstract naming. The logic is simple and the method is rough. There are some misnaming phenomena in the fields. In addition, some types of fields are too simple after abstract naming, and a lot of information is lost, which is not conducive to the use of information after log parsing. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method, system and electronic device for generating log field names based on a knowledge graph. The knowledge graph of the log data is obtained by processing sample log data, and then the names of the fields of the named logs are generated based on the knowledge graph. On the one hand, the fields can be named more accurately. On the other hand, more information can be obtained through the name of the field and the knowledge graph corresponding to the name in subsequent processing, which is more conducive to data aggregation, anomaly detection, root cause analysis, trend prediction, etc.

[0006] In order to achieve the above object, the first aspect of the present invention provides a method for generating log field names based on a knowledge graph, the method comprising the following steps:

[0007] Extracting attribute information and label information from the sample log data to establish an entity-relationship-attribute triple array of the sample log data;

[0008] Merge the ternary arrays of multiple sample log data to obtain the knowledge graph of the log data;

[0009] Get the log data to be named;

[0010] Extracting attribute information from the log data to be named, performing weighted matching on the attribute information of the log data to be named and the knowledge graph of the log data to obtain a matching result;

[0011] The names of the fields of the to-be-named log data are generated according to the matching results. The knowledge graph of the log data is obtained by processing the log data, and then the log fields are weighted matched based on the knowledge graph, and the names of the log fields are generated according to the matching results, thereby improving the accuracy of field naming.

[0012] Furthermore, the attribute information includes: location information, length information, format information, programming language information and log framework information;

[0013] The extracting attribute information includes:

[0014] Extracting a log header from the obtained sample log data or the to-be-named log data;

[0015] Extracting attribute information of each field from the log header;

[0016] The step of extracting tag information includes:

[0017] Extract the label information of each field from the log header of the sample log data. The log header is the main body of the log and is also the main object of log naming. The attribute information of each field extracted from the log header can effectively map the information of each field in the log header, which is conducive to establishing an effective knowledge graph of log data.

[0018] Furthermore, the step of establishing an entity-relationship-attribute triple array of sample log data includes:

[0019] Obtain the label information of each field in the log header of the sample log data as the entity, obtain each item in the attribute information as the relationship of the entity, obtain the attribute value of the attribute information as the attribute of each relationship, and establish the entity-relationship-attribute ternary array of each field of the sample log data;

[0020] Add a three-element array in each entity whose relation is the number of occurrences, and the initial attribute value is 1;

[0021] Merge the same entity in the sample log data and update the attribute value of the number of occurrences. The ternary array uses the tag information as the entity, which can reduce the number of entities and better reflect the essence of this field. The location information, length information, format information, programming language information, and log framework information as the associated information of the tag name can better represent the entity. Setting the number of occurrences as a statistical information can reflect the frequency of occurrence of the field in the log to facilitate the calculation of the frequency.

[0022] Further, extracting the log header includes:

[0023] Replace the fields in the fixed format in the sample log data or the log data to be named with the labeled fields to obtain the fields after preliminary labeling;

[0024] Determine the longest common substring of the sample log data or the log data to be named according to the initially labeled fields;

[0025] The part before the last space of the longest common substring is extracted as the log header of the sample log data or the log data to be named. By labeling the log data, it is easier to extract the longest common substring more accurately, so that a more accurate log header part can be obtained, and a more accurate knowledge graph can be established in subsequent processing.

[0026] Furthermore, extracting attribute information of each field includes:

[0027] Extracting the location information of each field from the log header of the sample log data or the log header of the log data to be named according to the location information extraction rule;

[0028] Extracting the length information of each field from the log header of the sample log data or the log header of the log data to be named according to the length information extraction rule;

[0029] Extracting format information of each field from the log header of the sample log data or the log header of the log data to be named according to the format tag matching;

[0030] Extracting programming language information of each field from a log header of sample log data or a log header of log data to be named according to programming rules;

[0031] Extracting log frame information of each field from the log header of the sample log data or the log header of the log data to be named according to the log frame matching;

[0032] The extracting tag information includes:

[0033] The label name information of each field is extracted from the log header of the sample log data according to the marked label name. The above information of each field extracted from the log header can fully describe each field.

[0034] Furthermore, the ternary arrays of the plurality of sample log data are fused to obtain a knowledge graph of the log data, including:

[0035] Extract the label name and attribute value of each triple array, and perform query matching in the existing knowledge graph;

[0036] If there is already a graph that is identical to the ternary array, then add the attribute value of the number of occurrences in the graph to the attribute value of the number of occurrences in the ternary array, and update the attribute value of the number of occurrences in the graph;

[0037] If there is no graph identical to the triple array, a new graph is created and the entire triple array is entered as a new graph. The knowledge graph is constructed by query matching and updating, and the constructed knowledge graph information is more comprehensive.

[0038] Furthermore, the weighted matching of the attribute information of the log data to be named with the knowledge graph of the log data to obtain a matching result includes:

[0039] Matching and querying the attribute value group corresponding to the attribute information of the to-be-named log data in the knowledge graph of the log data;

[0040] If a graph is found that all attribute values ​​are successfully matched, the label name corresponding to the attribute value group is returned as a suggestion option for field name generation;

[0041] If no graph is found that matches all attribute values ​​successfully, weighted matching is performed on the attribute value groups:

[0042] If the weighted match is successful, at least one tag name is returned as a suggested option for field name generation;

[0043] If the weighted matching fails, the message "no matching result" is returned. In the matching process, all attribute values ​​are matched first, which is more conducive to matching a more accurate map with the least steps. If all attribute values ​​fail to match, weighted matching is performed, which greatly reduces the need for weighted matching and saves system computing resources.

[0044] Furthermore, the method further comprises:

[0045] If the weighted matching fails, the label name of each field in the log header of the log data to be named is marked;

[0046] Extracting attribute information from the annotated log header and constructing an entity-relationship-attribute triple array of the log data to be named;

[0047] The knowledge graph is updated according to the ternary array of the log data to be named. The knowledge graph is continuously improved according to the matching results, so that the established knowledge graph can be continuously updated with the development of information technology and can always meet the needs of field naming over time.

[0048] Furthermore, the weighted matching of the attribute information of the log data to be named with the knowledge graph of the log data includes:

[0049] According to the matching rules, the matching degree array of each attribute value for each entity in the knowledge graph is calculated to obtain the matching degree matrix;

[0050] Use the trained logical classifier to predict the matching matrix to obtain the prediction matrix;

[0051] Converting the prediction matrix into probabilities using a logistic regression method;

[0052] The graph corresponding to the maximum probability is obtained as the weighted matching successful graph. The attribute value group is weighted by the logical classifier. Through algorithm learning, the system can provide more accurate name generation suggestion options.

[0053] A second aspect of the present invention provides a log field name generation system based on a knowledge graph, the system comprising:

[0054] A log preprocessing unit, used for extracting a log header from the acquired sample log data or the log data to be named;

[0055] An information extraction unit, used to extract attribute information from a log header of sample log data or a log header of log data to be named, and obtain tag information from the log header of sample log data;

[0056] A ternary array establishment unit, used to establish an entity-relationship-attribute ternary array of the sample log data according to the attribute information and label information of the extracted sample log data;

[0057] A knowledge graph building unit, used to fuse ternary arrays of multiple sample log data as a knowledge graph of the log data;

[0058] The matching and naming unit is used to perform weighted matching on the attribute information of the log data to be named and the knowledge graph of the log data to obtain a matching result; and to generate the name of each field of the log data to be named according to the matching result. The system obtains the knowledge graph of the log data by processing the sample log data, and then performs weighted matching on the fields of the log data to be named based on the knowledge graph, and generates the name of the log field according to the matching result, thereby improving the accuracy of field naming.

[0059] Furthermore, the log preprocessing unit includes:

[0060] A labeling processing module is used to replace the fields in the fixed format in the sample log data or the log data to be named with the labeled fields to obtain the fields after preliminary labeling;

[0061] The longest common substring determination module is used to determine the longest common substring of the sample log data or the log data to be named according to the initially labeled fields;

[0062] The log header extraction module is used to extract the part before the last space of the longest common substring as the log header of the sample log data or the log data to be named. By labeling the log data, it is easier to extract the longest common substring more accurately, so that a more accurate log header part can be obtained, and then a more accurate knowledge graph can be established in subsequent processing.

[0063] Furthermore, the information extraction unit includes:

[0064] The tag name information extraction module is used to extract the tag name information of each field from the log header of the sample log data according to the marked tag name;

[0065] A location information extraction module is used to extract the location information of each field from the log header of the sample log data or the log header of the log data to be named according to the location information extraction rule;

[0066] A length information extraction module is used to extract the length information of each field from the log header of the sample log data or the log header of the log data to be named according to the length information;

[0067] A format information extraction module is used to extract format information of each field from the log header of the sample log data or the log header of the log data to be named according to the format tag matching;

[0068] A programming language information extraction module is used to extract programming language information of each field from the log header of the sample log data or the log header of the log data to be named according to programming rules;

[0069] The log frame information extraction module is used to extract the log frame information of each field from the log header of the sample log data or the log header of the log data to be named according to the log frame matching. Different information extraction modules extract different information of each field, and different information can work together to fully describe each field.

[0070] Furthermore, the matching naming unit includes:

[0071] The matching naming unit includes:

[0072] An accurate matching module is used to match the attribute value group corresponding to the attribute information of the log data to be named in the knowledge graph of the log data; when a graph with all attribute values ​​successfully matched is found, a label name corresponding to the attribute value group is returned as a field name generation suggestion option;

[0073] The weighted matching module is used to perform weighted matching on attribute value groups when no graph is found that matches all attribute values ​​successfully:

[0074] If the weighted match is successful, at least one tag name is returned as a suggested option for field name generation;

[0075] If the weighted match fails, the message "No match" is returned. Exact match can match a more accurate graph. When the exact match fails, weighted match can provide reference suggestions for field naming.

[0076] Furthermore, weighted matching is performed between the attribute information of the to-be-named log data and the knowledge graph of the log data, including:

[0077] According to the matching rules, the matching degree array of each attribute value for each entity in the knowledge graph is calculated to obtain the matching degree matrix;

[0078] Use the trained logical classifier to predict the matching matrix to obtain the prediction matrix;

[0079] Converting the prediction matrix into probabilities using a logistic regression method;

[0080] The graph corresponding to the maximum probability is obtained as the weighted matching successful graph. The attribute value group is weighted by the logical classifier. Through algorithm learning, the system can provide more accurate name generation suggestion options.

[0081] A third aspect of the present invention provides an electronic device, comprising:

[0082] Memory for storing computer programs;

[0083] A processor is used to run the computer program to execute the method for generating log field names based on the knowledge graph.

[0084] On the other hand, the present invention provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the method for generating log field names based on a knowledge graph.

[0085] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements a method for generating log field names based on a knowledge graph.

[0086] Through the above technical solution, log field labels and various attributes can be extracted and stored in a structured manner in the knowledge graph, solidifying the knowledge of the log field names, making storage and query more regular, and facilitating the provision of name generation suggestions and the accumulation and update of knowledge.

[0087] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0089] Figure 1 It is a flow chart of a method for generating log field names based on a knowledge graph provided by an embodiment of the present invention;

[0090] Figure 2 A log header flow chart of each sample log data extracted by a log field name generation method based on a knowledge graph provided by an embodiment of the present invention;

[0091] Figure 3 It is a flow chart of a log field name generation method based on a knowledge graph provided by an embodiment of the present invention for extracting attribute information and label information of a log header of each sample log data;

[0092] Figure 4 It is a flow chart of extracting the log header of each log data to be named by a log field name generation method based on a knowledge graph provided by an embodiment of the present invention;

[0093] Figure 5 It is a flow chart of a log field name generation method based on a knowledge graph provided by an embodiment of the present invention for extracting attribute information of a log header of each log data to be named;

[0094] Figure 6 A method for generating a log field name based on a knowledge graph provided by an embodiment of the present invention is a flow chart for weighted matching of information of log data to be named with the knowledge graph of the log data;

[0095] Figure 7 It is a block diagram of a log field name generation system based on a knowledge graph provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0096] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0097] Figure 1 FIG. 1 is a flow chart of a method for generating log field names based on a knowledge graph provided by an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0098] Step 1: Extract attribute information and label information from the sample log data to establish an entity-relationship-attribute triple array of the sample log data, including:

[0099] S11: Extract the log header from the acquired sample log data, such as Figure 2 As shown, including:

[0100] S1101: Replace the fixed-format fields in the sample log data with labeled fields to obtain initially labeled fields. In this embodiment, a regular expression or a similar method is usually used to replace the fixed-format fields in the log data with labeled fields. For example, a lexical and grammatical analyzer generated by a tool Flex&Bison can be used. The lexical and grammatical analyzer is based on regular expressions.

[0101] S1102: Determine the longest common substring of the sample log data according to the initially labeled fields. In this embodiment, the LCS algorithm is usually used to find the longest common substring of each labeled log data.

[0102] S1103: Extract the part before the last space of the longest common substring as the log header of the sample log data, and the part after the last space as the log message body, and label the log message body as a whole. The labeled log data can more accurately extract the longest common substring, so as to obtain a more accurate log header part, and then establish a more accurate knowledge graph in subsequent processing.

[0103] In some embodiments, generally, multiple types of log data within a time window are obtained as sample log data, such as operating system logs, middleware logs, application logs, etc.

[0104] S12: Extract attribute information of each field from the log header of the extracted sample log data, such as Figure 3 As shown, including:

[0105] S1201: Extract the position information of each field from the log header of the sample log data according to the position information extraction rule. In this embodiment, the position information extraction rule is: starting from the first field from the left of a log, extract the field position of each field from the left.

[0106] S1202: extracting the length information of each field from the log header of the sample log data according to the length information extraction rule. In this embodiment, according to the length information extraction rule, the length information of the field should be the length of the entire string of the field, and the length needs to include internal spaces.

[0107] S1203: extracting format information of each field from the log header of the sample log data according to format tag matching. In this embodiment, the format tag is a tag corresponding to each field obtained according to a technique such as a regular expression.

[0108] S1204: Extract programming language information of each field from the log header of the sample log data according to programming rules. The programming language may be Java, Python, etc.

[0109] S1205: extracting log framework information of each field from the log header of the sample log data according to log framework matching, the log framework information may be jul, log4j, log4j2, logback, etc. The above information of each field extracted from the log header can fully describe each field.

[0110] S13: Extracting the label information of each field from the log header of the extracted sample log data, specifically including: extracting the label name information of each field from the log header of the sample log data according to the marked label name. The label name information of the field is, for example: Date&Time, Level, Component, PID-TID, IP, DIR, etc. The format labels of some fields can basically be used as label names, such as Level (log message level), IP (network address), etc., and for example, PID-TID, its format may be identified as INT-INT, then its label name needs to be marked out, marking the label name is only to mark the sample log data, and the log data to be named is not marked with the label name, and the label name cannot be extracted.

[0111] S14: Establishing an entity-relationship-attribute triple array of sample log data based on the attribute information and label information of each field, including:

[0112] S1401: Obtain label information of each field in the log header of the sample log data as an entity, obtain each item in the attribute information as a relationship of the entity, obtain the attribute value of the attribute information as an attribute of each relationship, and establish an entity-relationship-attribute triple array of each field of the sample log data;

[0113] S1402: Add a ternary array in each entity whose relation is the number of occurrences, and the initial attribute value is 1;

[0114] S1403: Merge the same entity in the sample log data and update the attribute value of the number of occurrences. The ternary array uses the tag name as the entity, which can reduce the number of entities and better reflect the essence of this field. Position information, length information, format information, programming language information, and log framework information as the associated information of the tag name can better represent the entity. Setting the number of occurrences as a statistical information can reflect the frequency of occurrence of the field in the log to facilitate the calculation of the frequency.

[0115] Step 2: Merge the ternary arrays of multiple sample log data to obtain the knowledge graph of the log data, including:

[0116] S21: Extract the label name and attribute value of each triple array, and perform query matching in the existing knowledge graph; if a graph identical to the triple array already exists, jump to S22, otherwise jump to S23; it should be noted that in the query process of S21, there is no need to query the attribute value of the matching occurrence count.

[0117] S22: adding the attribute value of the number of occurrences in the graph to the attribute value of the number of occurrences in the ternary array, and updating the attribute value of the number of occurrences in the graph;

[0118] S23: Create a new graph, and enter all the triple arrays into the knowledge graph as a new graph. The knowledge graph is constructed by query matching and updating, and the constructed knowledge graph information is more comprehensive.

[0119] Step 3: Obtain the log data to be named. The log data to be named may be the channel log data uploaded by the associated system, or may be the unnamed log data already in the database.

[0120] Step 4: extracting attribute information from the log data to be named, performing weighted matching on the attribute information of the log data to be named and the knowledge graph of the log data to obtain a matching result, specifically including:

[0121] S41: extracting a log header from the acquired log data to be named, such as Figure 4 As shown, including:

[0122] S4101: Replace the fixed-format fields in the to-be-named log data with labeled fields to obtain initially labeled fields. In this embodiment, a regular expression or a similar method is usually used to replace the fixed-format fields in the log data with labeled fields. For example, a lexical and grammatical analyzer generated by a tool Flex&Bison can be used, and the lexical and grammatical analyzer is based on regular expressions.

[0123] S4102: Determine the longest common substring of the to-be-named log data according to the initially labeled fields. In this embodiment, an LCS algorithm is generally used to find the longest common substring of each labeled log data.

[0124] S4103: Extract the part before the last space of the longest common substring as the log header of the log data to be named, and the part after the last space as the log message body, and label the log message body as a whole. The labeled log data can more accurately extract the longest common substring, so as to obtain a more accurate log header part, and then establish a more accurate knowledge graph in subsequent processing.

[0125] S42: Extract attribute information of each field from the log header of the extracted log data to be named, such as Figure 5 As shown, including:

[0126] S4201: Extract the position information of each field from the log header of the log data to be named according to the position information extraction rule. In this embodiment, the position information extraction rule is: starting from the first field from the left of a log, extract the field position of each field from the left.

[0127] S4202: Extract the length information of each field from the log header of the log data to be named according to the length information extraction rule. In this embodiment, according to the length information extraction rule, the length information of the field should be the length of the entire string of the field, and the length needs to include internal spaces.

[0128] S4203: extracting format information of each field from the log header of the log data to be named according to format tag matching. In this embodiment, the format tag is a tag corresponding to each field obtained according to a technique such as a regular expression.

[0129] S4204: Extract programming language information of each field from the log header of the log data to be named according to programming rules. The programming language may be Java, Python, etc.

[0130] S4205: extracting log framework information of each field from the log header of the log data to be named according to log framework matching, the log framework information may be jul, log4j, log4j2, logback, etc. The above information of each field extracted from the log header can fully describe each field.

[0131] S43: Perform weighted matching on the attribute information of the log data to be named and the knowledge graph of the log data to obtain a matching result, such as Figure 6 As shown, including:

[0132] S4301: Match and search the knowledge graph of the log data for the attribute value group corresponding to the attribute information of the log data to be named; if all attribute values ​​are successfully matched in the query, jump to S4303, otherwise jump to S4302;

[0133] S4302: Perform weighted matching on the attribute value group. If the weighted matching succeeds, jump to S4303; otherwise, jump to S4304.

[0134] S4303: Return the tag name corresponding to the attribute value group as a field name generation suggestion option;

[0135] S4304: No matching results are returned.

[0136] In the matching process, all attribute values ​​are matched first, which is more conducive to matching a more accurate map with the least steps. When all attribute values ​​fail to match, weighted matching is performed. The demand for weighted matching is greatly reduced, saving the system's computing resources.

[0137] In this embodiment, weighted matching of attribute value groups includes:

[0138] According to the matching rules, the matching degree array of each attribute value for each entity in the knowledge graph is calculated to obtain a matching degree matrix. In this embodiment, some attribute values ​​must use exact matching, such as programming language information, log framework information, etc. The matching degree of such attribute values ​​is in the form of {0, 1}, that is, either 1 or 0; some attribute values ​​can use weighted matching such as length information, format information, etc. The matching degree of such attribute values ​​is set to a value between the interval [0, 1]. For example, for length, the matching degree can be set to the ratio of the minimum digit difference to the average and the remainder of 1. For example, for format, the matching degree can be set to the ratio of the minimum edit distance to the average length and the remainder of 1.

[0139] The matching degree matrix is ​​predicted using the trained logic classifier to obtain a prediction matrix. In the present embodiment, the logic classifier is implemented using the logic classifier of the machine learning platform TensorFlow. The logic classifier is a linear classifier that receives input (such as a matrix, etc.) and performs a linear function on the input to generate a prediction. The linear function is actually a matrix multiplication, which represents all inputs as a large matrix X, multiplied by the corresponding weight W matrix and then added to the bias matrix B to generate a prediction matrix. The weights and biases of the logic classifier are adjusted through the training process to minimize the error of the predicted result, that is, to make the probability of the correct classification very close to 1, and the probability of other classifications close to 0. The training of the logic classifier is the same as the existing classifier training method, which will not be repeated in this application.

[0140] The prediction matrix is ​​converted into probability using the logistic regression method Softmax method.

[0141] The graph corresponding to the maximum probability is obtained as the weighted matching successful graph. The attribute value group is weighted by the logical classifier. Through algorithm learning, the system can provide more accurate name generation suggestion options.

[0142] In some embodiments, Figure 6 As shown, step four also includes:

[0143] S44: if the weighted matching fails, marking the label name of each field in the log header of the log data to be named;

[0144] S45: extracting attribute information from the annotated log header, and constructing an entity-relationship-attribute triple array of the log data to be named;

[0145] S46: Update the knowledge graph according to the ternary array of the log data to be named. The knowledge graph is continuously improved according to the matching results, so that the established knowledge graph can be continuously updated with the development of information technology and can always meet the needs of field naming over time.

[0146] Step 5: Generate the names of the fields of the log data to be named based on the matching results.

[0147] This method obtains a knowledge graph of log data by processing log data, then performs weighted matching on log fields based on the knowledge graph, generates the names of log fields according to the matching results, and improves the accuracy of field naming.

[0148] Embodiment 1

[0149] In this embodiment, first, log data in a time window is obtained from sources such as operating system logs, middleware logs, and application logs. For example, about 2,000 logs in a period of time are obtained from the HDFS component and the file name is saved as HDFS.log.

[0150] Then, the lexical and grammatical analyzers generated by Flex&Bison are used to replace the fixed-format fields in the log data to obtain the initially labeled fields. For example, a line in HDFS.log "081109203615 148 INFOdfs.DataNode$PacketResponder:PacketResponder 1 for block blk_38865049064139660 terminating" can be parsed as " <int> <int> <int> <level> <component>:PacketResponder 1 for block blk_38865049064139660terminating”;

[0151] Then, use the LCS algorithm to process the labeled HDFS.log and find the longest common substring. In the above example, the longest common substring is " <int> <int> <int> <level> <component>:”;

[0152] Then the last space is used as the boundary to split the log entries in the above example. The log entries can be divided into " <int> <int> <int> <level> <component> : <content>”.

[0153] In actual processing, it is necessary to annotate some format labels of the labeled log header with label names. The above example after annotation is expressed as: <date> <time> <pid> <level> <component> : <content>", extract the tag name information, location information, length information, format information, programming language information and log framework information for the above instance, and use the tag <pid>For example, its attributes are:

[0154] Tag name: Pid; Position: 3; Length: 3; Format: Int; Programming language: Java; Log framework: log4j.

[0155] Take the tag name as the entity, the location information, length information, format information, programming language information and log framework information as the relationship between the entities, and the attribute values ​​of the location information, length information, format information, programming language information and log framework information as the attributes of each relationship to establish the entity-relationship-attribute ternary array of each log data. At the same time, add the number of occurrences attribute with an initial value of 1. Merge the same tag names in the HDFS.log file, accumulate the number of occurrences, and get the ternary array with the tag name as the entity name as follows:

[0156] Pid Location 3 Pid length 2-5 Pid Format Int Pid programming language Java Pid Logging framework log4j Pid Occurrences 2000

[0157] Then enter the ternary array to obtain the knowledge graph of the log field. According to the label name and attribute value of the ternary array of each field, query and match in the knowledge graph. If there is no Pid entity with the same attribute value, create this entity and enter it in the knowledge graph. If there is, add the number of occurrences of the two and update the attribute value of the number of occurrences in the graph, thus forming a knowledge graph of the log data.

[0158] When using the above knowledge graph to generate log field names, the first thing to do is to process the log data to be named. For example, a new log: "081111 102017 26347 INFO dfs.DataNode$DataXceiver:Receivingblock blk_4343207286455274569 src: / 10.250.9.207:59759dest: / 10.250.9.207:50010", after preliminary labeling, it becomes: " <int> <int> <int> <level> <component>:Receiving block blk_4343207286455274569 src: / 10.250.9.207:59759dest: / 10.250.9.207:50010",

[0159] The log header is divided into: <int> <int> <int> <level> <component>:", with the third <int>For example, extract its attributes as follows:

[0160] Position: 3; Length: 6; Format: Int; Programming language: Java; Logging framework: log4j.

[0161] Match the above attribute value groups in the knowledge graph and query the entities that match <pid>, graph output <pid>, as here <int>The tag name of the .

[0162] If the extracted attributes are: position: 3; length: 6; format: Int; programming language: Java; log framework: log4j, and there is no entity that fully matches the query, then calculate the matching degree array of the above attribute values ​​for each entity in the graph to form a matching degree matrix X, use the weight matrix W and bias matrix B generated by training to generate a prediction matrix, and use the Softmax method to convert it into probability. For example, the above attributes finally get probabilities (0.7 0.1 0.15 0.05), where the probability of 0.7 corresponds to the entity <pid>, take the label name of the corresponding entity with the highest probability, that is <pid>Generate output as suggested name.

[0163] Figure 7 1 is a block diagram of a system for generating log field names based on a knowledge graph provided by an embodiment of the present invention. Figure 7 As shown, the system comprises:

[0164] A log preprocessing unit, used for extracting a log header from the acquired sample log data or the log data to be named;

[0165] An information extraction unit, used to extract attribute information from a log header of sample log data or a log header of log data to be named, and obtain tag information from the log header of sample log data;

[0166] A ternary array establishment unit, used to establish an entity-relationship-attribute ternary array of the sample log data according to the attribute information and label information of the extracted sample log data;

[0167] A knowledge graph building unit, used to fuse ternary arrays of multiple sample log data to obtain a knowledge graph of the log data;

[0168] The matching and naming unit is used to perform weighted matching on the attribute information of the log data to be named and the knowledge graph of the log data to obtain a matching result; and to generate the name of each field of the log data to be named according to the matching result. The system obtains the knowledge graph of the log data by processing the log data, and then performs weighted matching on the log fields based on the knowledge graph, and generates names for the log fields according to the matching results, thereby improving the accuracy of field naming.

[0169] In one embodiment, the log preprocessing unit includes:

[0170] The labeling processing module is used to replace the fixed-format fields in the sample log data or the to-be-named log data with the labeled fields to obtain the initially labeled fields. In this embodiment, a regular expression or a similar method is usually used to replace the fixed-format fields in the log data with labeled fields. For example, a lexical and grammatical analyzer generated by the tool Flex&Bison can be used, and the lexical and grammatical analyzer is based on regular expressions.

[0171] The longest common substring determination module is used to determine the longest common substring of the sample log data or the to-be-named log data according to the initially labeled fields. In this embodiment, the LCS algorithm is usually used to find the longest common substring of each labeled log data.

[0172] The log header extraction module is used to extract the part before the last space of the longest common substring as the log header of the sample log data or the log data to be named. By labeling the log data, it is easier to extract the longest common substring more accurately, so that a more accurate log header part can be obtained, and then a more accurate knowledge graph can be established in subsequent processing.

[0173] In some embodiments, the information extraction unit includes:

[0174] The tag name information extraction module is used to extract the tag name information of each field from the log header of the sample log data according to the marked tag name; as the sample log data, its tag name needs to be manually marked.

[0175] The position information extraction module is used to extract the position information of each field from the log header of the sample log data or the log header of the log data to be named according to the position information extraction rule. In this embodiment, the position information extraction rule is: starting from the first field from the left of a log, extract the field position of each field from the left.

[0176] The length information extraction module is used to extract the length information of each field from the log header of the sample log data or the log header of the log data to be named based on the length information. In this embodiment, according to the length information extraction rule, the length information of the field should be the length of the entire string of the field, and the length needs to include internal spaces.

[0177] The format information extraction module is used to extract the format information of each field from the log header of the sample log data or the log header of the log data to be named according to the format label matching. In this embodiment, the format label is the label corresponding to each field obtained based on regular expression and other technologies.

[0178] The programming language information extraction module is used to extract the programming language information of each field from the log header of the sample log data or the log header of the log data to be named according to the programming rules. The programming language can be Java, Python, etc.

[0179] The log framework information extraction module is used to extract the log framework information of each field from the log header of the sample log data or the log header of the to-be-named log data according to the log framework matching. The log framework information can be jul, log4j, log4j2, logback, etc. Different information extraction modules extract different information of each field, and different information can work together to fully describe each field.

[0180] Optionally, the matching naming unit includes:

[0181] An accurate matching module is used to match the attribute value group corresponding to the attribute information of the log data to be named in the knowledge graph of the log data; when a graph with all attribute values ​​successfully matched is found, a label name corresponding to the attribute value group is returned as a field name generation suggestion option;

[0182] The weighted matching module is used to perform weighted matching on attribute value groups when no graph is found that matches all attribute values ​​successfully:

[0183] If the weighted match is successful, at least one tag name is returned as a suggested option for field name generation;

[0184] If the weighted match fails, the message "No match" is returned. Exact match can match a more accurate graph. When the exact match fails, weighted match can provide reference suggestions for field name generation.

[0185] In this embodiment, weighted matching is performed between the attribute information of the to-be-named log data and the knowledge graph of the log data, including:

[0186] According to the matching rules, the matching degree array of each attribute value for each entity in the knowledge graph is calculated to obtain the matching degree matrix;

[0187] Use the trained logical classifier to predict the matching matrix to obtain the prediction matrix;

[0188] Converting the prediction matrix into probabilities using a logistic regression method;

[0189] The graph corresponding to the maximum probability is obtained as the weighted matching successful graph. The attribute value group is weighted by the logical classifier. Through algorithm learning, the system can provide more accurate name generation suggestion options.

[0190] The present invention also provides an electronic device, comprising:

[0191] Memory for storing computer programs;

[0192] A processor is used to run the computer program to execute the method for generating log field names based on the knowledge graph.

[0193] On the other hand, the present invention provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the method for generating log field names based on a knowledge graph.

[0194] On the other hand, the present invention provides a computer program product, including a computer program, which implements the method for generating log field names based on the knowledge graph when executed by a processor.

[0195] Those skilled in the art will understand that all or part of the steps in the method for implementing the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions for making a single-chip microcomputer, a chip or a processor (processor) perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0196] The optional embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the technical concept of the embodiments of the present invention, the technical scheme of the embodiments of the present invention can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the embodiments of the present invention. It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the embodiments of the present invention will not further describe various possible combinations.

[0197] In addition, various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed in the embodiments of the present invention.< / pid> < / pid> < / int> < / pid> < / pid> < / int> < / component> < / level> < / int> < / int> < / int> < / component> < / level> < / int> < / int> < / int> < / pid> < / content> < / component> < / level> < / pid> < / time> < / date> < / content> < / component> < / level> < / int> < / int> < / int> < / component> < / level> < / int> < / int> < / int> < / component> < / level> < / int> < / int> < / int>

Claims

1. A method for generating log field names based on knowledge graph, characterized in that: The method comprises the following steps: Extracting attribute information and label information from the sample log data to establish an entity-relationship-attribute triple array of the sample log data; Merge the ternary arrays of multiple sample log data to obtain the knowledge graph of the log data; Get the log data to be named; Extracting attribute information from the log data to be named, performing weighted matching on the attribute information of the log data to be named and the knowledge graph of the log data to obtain a matching result; Generate the name of each field of the to-be-named log data according to the matching result; The weighted matching of the attribute information of the log data to be named with the knowledge graph of the log data to obtain a matching result includes: Matching and querying the attribute value group corresponding to the attribute information of the to-be-named log data in the knowledge graph of the log data; If a graph is found that all attribute values ​​are successfully matched, the label name corresponding to the attribute value group is returned as a suggestion option for field name generation; If no graph is found that matches all attribute values ​​successfully, weighted matching is performed on the attribute value groups: If the weighted match is successful, at least one tag name is returned as a suggested option for field name generation; If the weighted matching fails, a message indicating that there is no matching result is returned.

2. The method according to claim 1, characterized in that The attribute information includes: location information, length information, format information, programming language information and log framework information; The extracting attribute information includes: Extracting a log header from the obtained sample log data or the to-be-named log data; Extracting attribute information of each field from the log header; The step of extracting tag information includes: Extract the label information of each field from the log header of the sample log data.

3. The method according to claim 2, characterized in that The step of establishing an entity-relationship-attribute triple array of sample log data includes: Obtain the label information of each field in the log header of the sample log data as the entity, obtain each item in the attribute information as the relationship of the entity, obtain the attribute value of the attribute information as the attribute of each relationship, and establish the entity-relationship-attribute ternary array of each field of the sample log data; Add a three-element array in each entity whose relation is the number of occurrences, and the initial attribute value is 1; Merge the same entity in the sample log data and update the attribute value of the number of occurrences.

4. The method according to claim 2, characterized in that: The step of extracting the log header includes: Replace the fields in the fixed format in the sample log data or the log data to be named with the labeled fields to obtain the fields after preliminary labeling; Determine the longest common substring of the sample log data or the log data to be named according to the initially labeled fields; The portion before the last space of the longest common substring is extracted as the log header of the sample log data or the log data to be named.

5. The method according to claim 2, characterized in that: The extracting of attribute information of each field includes: Extracting the location information of each field from the log header of the sample log data or the log header of the log data to be named according to the location information extraction rule; Extracting the length information of each field from the log header of the sample log data or the log header of the log data to be named according to the length information extraction rule; Extracting format information of each field from the log header of the sample log data or the log header of the log data to be named according to the format tag matching; Extracting programming language information of each field from a log header of sample log data or a log header of log data to be named according to programming rules; Extracting log frame information of each field from the log header of the sample log data or the log header of the log data to be named according to the log frame matching; The extracting tag information includes: The tag name information of each field is extracted from the log header of the sample log data according to the marked tag name.

6. The method according to claim 1, characterized in that The method of fusing multiple sample log data to obtain a knowledge graph of the log data includes: Extract the label name and attribute value of each triple array and perform query matching in the existing knowledge graph; If there is already a graph that is identical to the ternary array, then add the attribute value of the number of occurrences in the graph to the attribute value of the number of occurrences in the ternary array, and update the attribute value of the number of occurrences in the graph; If there is no map identical to the ternary array, a new map is created and the entire ternary array is entered as a new map.

7. The method according to claim 1, characterized in that The method further comprises: If the weighted matching fails, the label name of each field in the log header of the log data to be named is marked; Extracting attribute information from the annotated log header and constructing an entity-relationship-attribute triple array of the log data to be named; Update the knowledge graph according to the ternary array of the log data to be named.

8. The method according to claim 1, characterized in that The weighted matching of the attribute information of the to-be-named log data with the knowledge graph of the log data includes: According to the matching rules, the matching degree array of each attribute value for each entity in the knowledge graph is calculated to obtain the matching degree matrix; Use the trained logical classifier to predict the matching matrix to obtain the prediction matrix; Converting the prediction matrix into probabilities using a logistic regression method; The spectrum corresponding to the maximum probability is obtained as the spectrum with successful weighted matching.

9. A log field name generation system based on knowledge graph, characterized in that: The system comprises: A log preprocessing unit, used for extracting a log header from the acquired sample log data or the log data to be named; An information extraction unit, used to extract attribute information from a log header of sample log data or a log header of log data to be named, and obtain tag information from the log header of sample log data; A ternary array establishment unit, used to establish an entity-relationship-attribute ternary array of the sample log data according to the attribute information and label information of the extracted sample log data; A knowledge graph building unit, used to fuse ternary arrays of multiple sample log data to obtain a knowledge graph of the log data; A matching and naming unit, used for performing weighted matching on the attribute information of the to-be-named log data and the knowledge graph of the log data to obtain a matching result; and generating the name of each field of the to-be-named log data according to the matching result; The matching naming unit includes: An accurate matching module is used to match the attribute value group corresponding to the attribute information of the log data to be named in the knowledge graph of the log data; when a graph with all attribute values ​​successfully matched is found, a label name corresponding to the attribute value group is returned as a field name generation suggestion option; The weighted matching module is used to perform weighted matching on attribute value groups when no graph is found that matches all attribute values ​​successfully: If the weighted match is successful, at least one tag name is returned as a suggested option for field name generation; If the weighted matching fails, a message indicating that there is no matching result is returned.

10. The system according to claim 9, characterized in that The log preprocessing unit comprises: A labeling processing module is used to replace the fields in the fixed format in the sample log data or the log data to be named with the labeled fields to obtain the fields after preliminary labeling; The longest common substring determination module is used to determine the longest common substring of the sample log data or the log data to be named according to the initially labeled fields; The log header extraction module is used to extract the part before the last space of the longest common substring as the log header of the sample log data or the log data to be named.

11. The system according to claim 9, characterized in that The information extraction unit comprises: The tag name information extraction module is used to extract the tag name information of each field from the log header of the sample log data according to the marked tag name; A location information extraction module is used to extract the location information of each field from the log header of the sample log data or the log header of the log data to be named according to the location information extraction rule; A length information extraction module is used to extract the length information of each field from the log header of the sample log data or the log header of the log data to be named according to the length information; A format information extraction module is used to extract format information of each field from the log header of the sample log data or the log header of the log data to be named according to the format tag matching; A programming language information extraction module is used to extract programming language information of each field from the log header of the sample log data or the log header of the log data to be named according to programming rules; The log frame information extraction module is used to extract the log frame information of each field from the log header of the sample log data or the log header of the log data to be named according to the log frame matching.

12. The system according to claim 9, characterized in that The attribute information of the to-be-named log data is weightedly matched with the knowledge graph of the log data, including: According to the matching rules, the matching degree array of each attribute value for each entity in the knowledge graph is calculated to obtain the matching degree matrix; Use the trained logical classifier to predict the matching matrix to obtain the prediction matrix; Converting the prediction matrix into probabilities using a logistic regression method; The spectrum corresponding to the maximum probability is obtained as the spectrum with successful weighted matching.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to run the computer program to execute the log field name generation method based on the knowledge graph described in any one of claims 1-8.

14. A machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the log field name generation method based on a knowledge graph as described in any one of claims 1-8.

15. A computer program product comprising a computer program, characterized in that When executed by a processor, the computer program implements the log field name generation method based on a knowledge graph as described in any one of claims 1 to 8.

Images