Log data processing method

Abstract

The invention provides a log data processing method, and relates to the technical field of big data. The method comprises the following steps: acquiring log data marked with error information, and extracting a plurality of key fields of the log data; a preset rule set is obtained, a multi-way tree structure constructed according to the rule set is used for matching each key field, a matching result is obtained, and the matching result represents whether the key fields are matched with error information or not; when the matching results of the plurality of key fields fail, reasoning a plurality of association paths of the log data by using a pre-generated knowledge graph; and screening the plurality of associated paths based on the pre-estimated path weight to obtain a propagation path of the log data so as to locate error information. The invention further provides a log data processing device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and in particular to a method, device, equipment, medium and program product for processing log data. Background Art

[0002] At present, with the rapid development of big data technology, the banking industry log error diagnosis system has gradually achieved a leap from simple rule matching to complex pattern recognition. Most systems have been able to realize the automatic collection, classification and storage of log data, and use algorithm models to warn and locate potential errors.

[0003] However, due to the complexity of banking operations and the massive amount of log data, the existing system still needs to be further improved in terms of error diagnosis accuracy, real-time performance, and user experience. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a method, apparatus, device, medium and program product for processing log data, which can efficiently perform error diagnosis and decision making on log data.

[0005] According to a first aspect of the present disclosure, a method for processing log data is provided, including: obtaining log data marked with error information, and extracting multiple key fields of the log data; obtaining a preset rule set, and matching each key field using a multi-branch tree structure constructed according to the rule set to obtain a matching result, wherein the matching result represents whether the key field matches the error information; when the matching results of multiple key fields all fail, using a pre-generated knowledge graph to infer multiple associated paths of the log data; and screening the multiple associated paths based on the estimated path weights to obtain a propagation path of the log data to locate the error information.

[0006] According to an embodiment of the present disclosure, a polytree structure constructed based on a rule set includes: loading the rule set from a specified path and determining the file type of the rule set; calling a parser corresponding to the file type and converting the rule set into a polytree structure.

[0007] According to an embodiment of the present disclosure, a rule set includes multiple rules for identifying log data errors, each rule including at least one field condition, an error code and a rule description; converting the rule set into a polytree structure includes: parsing the rule set into structured data; initializing the root node of the polytree structure; traversing each rule of the rule set of structured data, and inserting each field condition into the child node of the root node one by one according to whether each field condition of the rule exists in the polytree structure; when it is determined that the traversal of the rule is completed, storing the error code and rule description of the rule in the leaf node of the child node, and returning to the step of inserting each field condition into the child node of the root node one by one for the next rule of the rule.

[0008] According to an embodiment of the present disclosure, a multi-branch tree structure constructed according to a rule set is used to match each key field to obtain a matching result, including: for each key field, a fuzzy matching algorithm is used to perform a first match between a leaf node of the multi-branch tree structure and the key field; when the results of the first match of multiple key fields all fail, a multi-field joint matching algorithm is used to perform a second match between the multi-branch tree structure and multiple key fields; when the results of the second match fail, it is determined that the matching results of multiple key fields all fail.

[0009] According to an embodiment of the present disclosure, a multi-field joint matching algorithm is used to perform a second match between a multi-branch tree structure and multiple key fields, including: for any target path from the root node to the leaf node of the multi-branch tree structure, multiple key fields are matched level by level according to the hierarchical structure of the multi-branch tree structure to obtain matching degrees at each level; a corresponding weighted score is assigned to each key field; weighted summation of the matching degrees at each level and the corresponding weighted scores is performed to obtain a total matching degree between the multi-branch tree structure and the multiple key fields; it is determined whether the total matching degree is greater than or equal to a preset matching degree threshold, and if so, it is determined that the result of the second match is successful; otherwise, for another target path of the multi-branch tree structure, the step of matching multiple key fields level by level is returned.

[0010] According to an embodiment of the present disclosure, a knowledge graph is pre-generated in the following manner: obtaining source log data that has been error-processed and its corresponding error causes; integrating and optimizing the source log data and the error causes to generate a knowledge graph, wherein integrating and optimizing the source log data and the error causes includes: configuring multiple entities, entity attributes, and entity relationships according to the source log data and the error causes, wherein the entities represent objects in the source log data, and the entity relationships represent the association relationships between the entities; for any two entities, calculating the first similarity between the corresponding two entity attributes; when the first similarity is greater than a set first similarity threshold, merging the two entities; for any two entity relationships, calculating the second similarity between the corresponding two relationship attributes; when the second similarity is greater than a set second similarity threshold, merging the two entity relationships.

[0011] According to an embodiment of the present disclosure, before configuring multiple entities, entity attributes, and relationships between entities, it also includes: preprocessing source log data and error causes, wherein the preprocessing includes missing value processing, outlier cleaning, and duplicate data removal.

[0012] According to an embodiment of the present disclosure, a pre-generated knowledge graph is used to infer multiple associated paths of log data, including: using set inference rules to infer the log data, wherein the inference rules include entity matching rules and path search rules; and determining multiple associated paths of the log data based on the inference results.

[0013] A second aspect of the present disclosure provides a log data processing device, including: a field extraction module, used to obtain log data marked with error information, and extract multiple key fields of the log data; a field matching module, used to obtain a preset rule set, and use a multi-branch tree structure constructed according to the rule set to match each key field to obtain a matching result, wherein the matching result represents whether the key field matches the error information; a knowledge graph reasoning module, used to use a pre-generated knowledge graph to infer multiple associated paths of the log data when the matching results of multiple key fields all fail; an error location module, used to screen multiple associated paths based on estimated path weights, obtain a propagation path of the log data, and locate the error information.

[0014] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0015] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0016] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above contents and other purposes, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0018] Figure 1 The application scenario diagram schematically shows the method, apparatus, device, medium and program product for processing log data according to the embodiments of the present disclosure;

[0019] Figure 2 A flowchart schematically shows a method for processing log data according to an embodiment of the present disclosure;

[0020] Figure 3 A schematic diagram showing a principle diagram of converting a rule set into a multi-branch tree structure according to an embodiment of the present disclosure is shown;

[0021] Figure 4 A flowchart of converting a rule set into a multi-branch tree structure according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 5 A flowchart of key field matching according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 6 The schematic diagram shows a principle diagram of integrating and optimizing source log data and error causes according to an embodiment of the present disclosure;

[0024] Figure 7 A block diagram schematically shows a log data processing device according to an embodiment of the present disclosure;

[0025] Figure 8 A block diagram of an electronic device suitable for implementing a method for processing log data according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0027] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0028] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0029] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0030] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0032] As can be seen from the background technology, the current log data processing faces the challenges of large data volume, high complexity, and high standards for real-time performance and accuracy.

[0033] In view of this, the present disclosure provides a method for processing log data, which relates to the field of big data technology. The method includes: obtaining log data marked with error information, extracting multiple key fields of the log data; obtaining a preset rule set, using a multi-branch tree structure constructed according to the rule set to match each key field, and obtaining a matching result, wherein the matching result characterizes whether the key field matches the error information; when the matching results of multiple key fields all fail, using a pre-generated knowledge graph to infer multiple associated paths of the log data; based on the estimated path weights, multiple associated paths are screened to obtain the propagation path of the log data to locate the error information. The present disclosure can efficiently perform error diagnosis and decision-making on log data.

[0034] Figure 1 The application scenario diagram of the log data processing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0035] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0036] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0037] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0038] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0039] It should be noted that the log data processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the log data processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The log data processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the log data processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0040] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0041] The following will be based on Figure 1 The scene described by Figure 2~Figure 6 The method for processing log data in an embodiment of the present disclosure is described in detail.

[0042] Figure 2 The flowchart schematically shows a method for processing log data according to an embodiment of the present disclosure.

[0043] like Figure 2 As shown, the log data processing method of this embodiment includes operations S210 to S240, and the log data processing method can be executed by the server.

[0044] In operation S210 , log data with error information is acquired, and a plurality of key fields of the log data are extracted.

[0045] For example, from the cleaned log data cluster, read the log data marked with error information, such as timestamp, error code, log level, service module, message content, etc. Then, extract multiple key fields, such as timestamp, error code, message content, etc.

[0046] In operation S220, a preset rule set is obtained, and each key field is matched using a multi-branch tree structure constructed according to the rule set to obtain a matching result, wherein the matching result indicates whether the key field matches error information.

[0047] In this embodiment, the rule set includes multiple rules for identifying error information in log data, and the multiple rules can be defined in a rule set file using a predetermined file type. Each rule includes at least one field condition, an error code, and a rule description. The field condition can be a rule name, a matching pattern (such as a regular expression), etc.

[0048] In operation S230 , when the matching results of multiple key fields all fail, multiple association paths of the log data are inferred using a pre-generated knowledge graph.

[0049] For example, the server may first use a multi-tree structure to perform field matching and when the matching fails, it may then use a knowledge graph for processing to obtain multiple associated paths of the log data.

[0050] In operation S240 , multiple associated paths are screened based on the estimated path weights to obtain a propagation path of the log data to locate the error information.

[0051] For example, after obtaining multiple associated paths, at least one propagation path may be screened out based on the estimated path weights to locate error information in the log data.

[0052] Through the embodiments of the present disclosure, error information in log data can be quickly and accurately identified and processed, improving the processing efficiency and accuracy of log data. In addition, by utilizing the powerful query and analysis capabilities of knowledge graphs and graph databases, efficient reasoning and analysis of complex log data can be performed, providing strong support for subsequent fault location and decision-making.

[0053] In this embodiment, the multi-branch tree structure constructed according to the rule set in the above operation S220 includes: loading the rule set from a specified path and determining the file type of the rule set; calling a parser corresponding to the file type and converting the rule set into a multi-branch tree structure.

[0054] For example, the file type of the rule set is predetermined, and a parser corresponding to the file type of the rule set is used to convert the rule set into a multi-branch tree structure that can be quickly searched by a computer, so that fields and matching rules can be quickly retrieved. For complex rules involving multiple fields or conditions, logical operators (such as AND, OR, NOT) can be used for combined processing.

[0055] Figure 3 The schematic diagram shows a principle diagram of converting a rule set into a multi-branch tree structure according to an embodiment of the present disclosure. Figure 4 The flowchart of converting a rule set into a multi-branch tree structure according to an embodiment of the present disclosure is schematically shown.

[0056] like Figure 3 and Figure 4 As shown, in this embodiment, converting the rule set into a multi-branch tree structure includes operations S401 to S404.

[0057] In operation S401 , a rule set is parsed into structured data.

[0058] The above parser can be used to parse the rule set into structured data, such as a dictionary type.

[0059] In operation S402, a root node of a multi-branch tree structure is initialized.

[0060] In operation S403, each rule of the rule set of structured data is traversed, and each field condition is inserted into the child node of the root node one by one according to whether each field condition of the rule exists in the multi-branch tree structure.

[0061] For example, traverse each rule in the rule set and insert each field into the multitree structure. If the field already exists in the multitree, go down along the existing path; if the field does not exist in the multitree, create a new node in the multitree. Then, move to the next field node until the field traversal is completed.

[0062] In operation S404, when it is determined that the traversal of the rule is completed, the error code and rule description of the rule are stored in the leaf node of the child node, and for the next rule of the rule, return to the above operation S403 to insert each field condition into the child node of the root node one by one.

[0063] By analogy, all the rules of the rule base can be inserted into the multi-branch tree structure field by field, so that the fields and matching rules can be quickly retrieved.

[0064] Figure 5 The flowchart of key field matching according to an embodiment of the present disclosure is schematically shown.

[0065] like Figure 5 As shown, in this embodiment, the above operation S220 uses a multi-branch tree structure constructed according to a rule set to match each key field to obtain a matching result, including operations S521 to S523.

[0066] In operation S521, for each key field, a fuzzy matching algorithm is used to perform a first match between the leaf nodes of the multi-branch tree structure and the key field.

[0067] For the key fields of the log data, the fuzzy matching algorithm is first used to match the specific error information, and then the successfully matched log data is output. When the first match is successful, that is, the key field matches a certain rule, the log data can be marked as an error log and the relevant metadata (such as error code, occurrence time, log level, etc.) can be recorded.

[0068] In operation S522, when the results of the first matching of the multiple key fields all fail, a multi-field joint matching algorithm is used to perform a second matching between the multi-tree structure and the multiple key fields.

[0069] When the error information cannot be found in the log data through the fuzzy matching algorithm, the multi-field joint matching algorithm is further used for error matching. Similarly, when the second match is successful, the matching result is output, the log data is marked as an error log, and the relevant metadata (such as error code, occurrence time, log level, etc.) is recorded.

[0070] In operation S523, when the result of the second matching fails, it is determined that the matching results of the plurality of key fields all fail.

[0071] In this embodiment, the operation S522 uses a multi-field joint matching algorithm to perform a second match between the multi-tree structure and multiple key fields, including:

[0072] For any target path from the root node to the leaf node of the multi-branch tree structure, multiple key fields are matched level by level according to the hierarchical structure of the multi-branch tree structure to obtain the matching degree at each level;

[0073] Assign a corresponding weighted score to each key field;

[0074] Perform weighted summation of the matching degrees at each level and the corresponding weighted scores to obtain the total matching degree between the multi-branch tree structure and multiple key fields;

[0075] Determine whether the total matching degree is greater than or equal to a preset matching degree threshold. If so, determine that the result of the second matching is successful; otherwise, for another target path of the multi-branch tree structure, return to the step of matching multiple key fields level by level.

[0076] For example, the multiple key fields of a log data include: log level; error message, usually including error description; service name that generates the log. The execution steps of the multi-field joint matching algorithm can be:

[0077] Step 1: Get a piece of log data and parse its key fields, including log level, error message, and service name.

[0078] Step 2: Start matching from the root node of the multi-branch tree structure and check whether the value of the log level matches the value of the current node.

[0079] Step 3: If it matches, continue to check the value of the next field, which is the error message, until a leaf node is matched.

[0080] Step 4: If all key fields match, the second matching Chen Gong is determined, and the corresponding error code and error description are output. If a key field does not match, the other branches of the multi-branch tree structure are backtracked.

[0081] Then, each key field has a weighted score in the matching process in the multi-branch tree, and the weighted scores corresponding to each level of matching degree can be set, and the weighted scores can also be adjusted according to actual needs.

[0082] For example, in the multiple key fields of the above log data, the weighted scores corresponding to the matching degrees of log level, error message and service name are 0.3, 0.4 and 0.3 respectively, and the sum of the matching degrees at each level is 1. Finally, the total matching degree final_score between the multi-tree structure and multiple key fields is:

[0083] final_score=w(logLevel)×0.3+w(message)×0.4+w(service)×0.5

[0084] Among them, w(logLevel), w(message), and w(service) are the matching degrees of log level, error message, and service name respectively.

[0085] Whether the result of the second match is successful is determined based on the final total match score. For example, the preset match score threshold is 0.7, and if the total match score final_score is greater than or equal to 0.7, the result of the second match is determined to be successful.

[0086] Through the embodiments of the present disclosure, the multi-field joint matching algorithm is combined with the multi-branch tree structure, and the matching process of key fields in the log data is guided by the hierarchical structure of the multi-branch tree. Each layer of the multi-branch tree is regarded as a part of the field matching, and the matching conditions are gradually refined, and finally the leaf nodes are matched according to the log field values. In addition, the error information in the log data can be accurately identified and processed to support subsequent fault location and decision-making.

[0087] When the second match still fails, it can be considered that the field matching using the multi-tree structure has failed. The log data can be subsequently integrated and optimized and imported into the knowledge graph for further reasoning.

[0088] With the widespread application of knowledge graphs in the field of error diagnosis, graph databases provide powerful query and analysis capabilities, becoming an indispensable part of intelligent log diagnosis and decision-making systems.

[0089] According to an embodiment of the present disclosure, the knowledge graph is pre-generated in the following manner: obtaining source log data that has undergone error reporting processing and its corresponding error causes; integrating and optimizing the source log data and the error causes to generate a knowledge graph.

[0090] Figure 6 The schematic diagram shows a principle diagram for integrating and optimizing source log data and error causes according to an embodiment of the present disclosure.

[0091] like Figure 6 As shown, in this embodiment, the source log data and the error cause are integrated and optimized, including:

[0092] According to the source log data and the error cause, multiple entities, entity attributes, and relationships between entities are configured, where the entity represents the object in the source log data, and the relationship between entities represents the association relationship between entities;

[0093] For any two entities, calculate the first similarity between the corresponding two entity attributes;

[0094] When the first similarity is greater than a set first similarity threshold, the two entities are merged;

[0095] For any relationship between two entities, calculate the second similarity between the corresponding two relationship attributes;

[0096] When the second similarity is greater than a set second similarity threshold, the relationship between the two entities is merged.

[0097] Through the embodiments of the present disclosure, complex source log data and its error causes are integrated and optimized to generate a knowledge graph.

[0098] In some embodiments, after the source log data and its error causes are integrated and optimized, importing them into the knowledge graph may include: first, extracting knowledge graph entities and relationships related to the source log data from the graph database, wherein the entities represent specific objects in the source log data, and the relationships are used to describe the interconnections between entities. Secondly, in order to improve query efficiency, a full-text index is created to speed up the retrieval of entity attributes. Finally, in order to optimize system performance, frequently accessed entities (such as core services) and relationships (such as common call paths) are preloaded into memory, thereby reducing frequent queries to the graph database and speeding up subsequent access and query responses.

[0099] In this embodiment, before configuring multiple entities, entity attributes, and relationships between entities, it also includes: preprocessing the source log data and error causes, wherein the preprocessing includes missing value processing, outlier cleaning, and duplicate data removal.

[0100] For the source log data and error reasons, its missing values, abnormal values, etc. can be cleaned. For example, first, check the source log data to identify some problematic data, such as missing values, abnormal values, or erroneous data. Then, according to the set cleaning rules, process this part of the data to remove duplicate, incomplete, or erroneous data, thereby improving the quality of the source log data, ensuring the reliability and integrity of the source log data, and providing a reliable data foundation for subsequent analysis and application.

[0101] Then, according to the source log data and the error cause, multiple entities, entity attributes, and relationships between entities are configured. For example, the multiple entities configured may include a backend interface entity, a job group entity, a microservice entity, and an error entity.

[0102] Among them, the backend interface entity represents various APIs (application programming interfaces) and is responsible for receiving requests and providing services. The entity attributes of the backend interface entity include interface name, interface path, request method, response status code, request parameters, response data, etc. The related entity relationships include call relationships, which represent the call relationships between different interfaces.

[0103] The job group entity represents multiple job groups and is a logical collection of distributed computing tasks, usually used for large-scale data processing. The entity attributes of the job group entity include the job group name, job group code (ID), job group status, and the sub-job ID contained in the job group. The related inter-entity relationships include the inclusion relationship, which means that a job group contains multiple sub-jobs.

[0104] Microservice entities represent independently running microservices, and their entity attributes include service name, service ID, service path, service status, deployment location, etc.

[0105] The error entity represents the specific errors that may occur during the operation of the system in which the method is located and its related attributes, and is used to describe the identification of a specific error and its related information. The entity attributes of the error entity include error code, error description, error severity and timestamp.

[0106] After configuring multiple entities, continue to configure the relationship between entities. For example, the configured relationship between entities includes the backend interface call relationship, the microservice call relationship, the error information and the backend interface relationship, and the error information and the job group relationship.

[0107] Among them, the backend interface call relationship describes the call relationship between different interfaces, including call frequency, success rate and other indicators. The microservice call relationship defines the call relationship between microservices, including the call path, passed parameters and other information. The relationship between error information and backend interface refers to the establishment of a one-to-many relationship, indicating that a backend interface may trigger multiple error messages. The relationship between error information and job group refers to the establishment of a one-to-one relationship, indicating that an error message may only involve one job group.

[0108] Then, the configured multiple entities can be appropriately merged. For any two entities, the first similarity between the corresponding two entity attributes is calculated; when the first similarity is greater than the set first similarity threshold, the two entities are merged. Thus, entities with high similarity are merged or marked as the same entity. This can eliminate duplication between entities and ensure that each entity has a unique identifier.

[0109] For example, the same interface may be described in different ways in multiple source log data. By merging these duplicate entities, we can avoid the same entity existing multiple times in different forms, thereby reducing storage redundancy. In this way, the merged unique entity can uniformly describe all call information, which is convenient for problem tracing and error analysis. At the same time, after the entity is merged, the number of nodes in the knowledge graph is reduced, the structure is more concise and clear, and overly complex topological relationships are avoided.

[0110] For configured inter-entity relationships, relationship attributes and entity attributes can be used to integrate the relationships. For any two inter-entity relationships, the second similarity between the corresponding two relationship attributes is calculated; when the second similarity is greater than the set second similarity threshold, the two inter-entity relationships are merged.

[0111] That is to say, for the relationship between two entities, the common attributes and features of the two are extracted. Then, based on the similarity of the relationship between entities, the entity relationships between different data sources are merged or marked as the same entity relationship to ensure that the fused relationship can accurately reflect the actual situation. In this way, the complex associations between entities can be understood and managed more efficiently, providing reliable support for the analysis and decision-making of log data by knowledge graphs.

[0112] In this embodiment, a pre-generated knowledge graph is used to infer multiple associated paths of log data, including: using set reasoning rules to reason about the log data, wherein the reasoning rules include entity matching rules and path search rules; and determining multiple associated paths of the log data based on the reasoning results.

[0113] For example, first, the entity matching rules are used to match the entities in the error log with other entities in the knowledge graph to determine the association between them. The entity matching rules determine the relationship between the error log entity and other entities in the knowledge graph based on entity attributes and relationship attributes. Secondly, the path search rules are used to find multiple association paths between the error log entity and other entities in the knowledge graph.

[0114] Specifically, the entity matching rule can use string matching and semantic similarity calculation to determine the relationship between the error log entity and other entities in the knowledge graph. The path finding rule can use the shortest path algorithm and subgraph matching algorithm to find the shortest path between the error log entity and other entities and subgraphs with similar structures or attributes to discover possible root causes or solutions to similar situations.

[0115] For example, a graph query language can be used in the knowledge graph to obtain a specified log entity (such as Log123) and its directly or indirectly related neighbor nodes. Through this query, the 1-hop and 2-hop neighbor nodes related to the log entity Log123 are extracted, where the 1-hop neighbor node is an entity directly connected to Log123, and the 2-hop neighbor node is an entity connected to the 1-hop neighbor. Next, the reasoning context is built based on the extracted neighbor nodes. Taking Log123 as an example, the 1-hop neighbor is a service entity, and the 2-hop neighbor is the error code related to the service. The subgraph matching algorithm can identify and match the current log data with the known error paths in the knowledge graph, and extract multiple associated paths from the context information, thereby forming the knowledge background required for reasoning, providing basic data for subsequent knowledge graph reasoning operations.

[0116] Through the embodiments of the present disclosure, the information in the knowledge graph is deeply mined, and the intelligent diagnosis and reasoning of error logs are performed, which significantly improves the error troubleshooting and decision support capabilities of log data.

[0117] Finally, multiple associated paths are screened based on the estimated path weights to obtain the propagation path of the log data to locate the error information. Multiple associated paths can be screened by evaluating the path weights, and the obtained multiple associated paths can be effectively screened to determine which context information is valid and which information is invalid or has low relevance.

[0118] For example, the path weight of each associated path can be calculated by counting the call frequency and error frequency of multiple associated paths. Specifically, the calculation of path weight is based on the following two key factors: the call frequency indicates the frequency of the log entity appearing in the historical access records; the error frequency refers to the repeated appearance of a certain error path in the knowledge graph and its strong correlation with a specific log entity. Suppose there are two associated paths, path 1 appears 5 times, and path 2 appears 1 time. Since path 1 appears more frequently, its path weight is larger.

[0119] Through the embodiments of the present disclosure, in the case where there are multiple associated paths, the associated paths are sorted by path weight, and more reliable paths are screened out, helping users to intuitively understand the propagation path of error information in log data.

[0120] At this point, the following Table 1 gives the comparison results of the processing methods of using a multi-tree structure for field matching and using a knowledge graph for reasoning.

[0121] Table 1

[0122]

[0123] It can be seen that the combination of using a multi-tree structure for field matching and using a knowledge graph for reasoning can quickly and accurately identify and process error information in log data, improving the efficiency and accuracy of log data processing. In addition, by using the powerful query and analysis capabilities of knowledge graphs and graph databases, efficient reasoning and analysis of complex log data can provide strong support for subsequent fault location and decision-making.

[0124] Based on the above log data processing method, the present disclosure also provides a log data processing device. Figure 7 The device is described in detail.

[0125] Figure 7 A block diagram of a log data processing device according to an embodiment of the present disclosure is schematically shown.

[0126] like Figure 7 As shown, the log data processing device 700 of this embodiment includes a field extraction module 710, a field matching module 720, a knowledge graph reasoning module 730 and an error location module 740.

[0127] The field extraction module 710 is used to obtain log data marked with error information and extract multiple key fields of the log data. In one embodiment, the field extraction module 710 can be used to perform the operation S210 described above, which will not be described in detail here.

[0128] The field matching module 720 is used to obtain a preset rule set, and use a multi-branch tree structure constructed according to the rule set to match each key field to obtain a matching result, wherein the matching result indicates whether the key field matches the error information. In one embodiment, the field matching module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0129] The knowledge graph reasoning module 730 is used to use the pre-generated knowledge graph to reason about multiple association paths of the log data when the matching results of multiple key fields all fail. In one embodiment, the knowledge graph reasoning module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0130] The error location module 740 is used to screen multiple associated paths based on the estimated path weights to obtain the propagation path of the log data to locate the error information. In one embodiment, the error location module 740 can be used to perform the operation S240 described above, which will not be repeated here.

[0131] According to an embodiment of the present disclosure, the field matching module 720 is further used to: load a rule set from a specified path and determine a file type of the rule set; call a parser corresponding to the file type and convert the rule set into a multi-branch tree structure.

[0132] According to an embodiment of the present disclosure, a rule set includes multiple rules for identifying log data errors, each rule including at least one field condition, an error code and a rule description; the field matching module 720 is also used to: parse the rule set into structured data; initialize the root node of the multi-branch tree structure; traverse each rule of the rule set of structured data, and insert each field condition into the child node of the root node one by one according to whether each field condition of the rule exists in the multi-branch tree structure; when it is determined that the traversal of the rule is completed, the error code and rule description of the rule are stored in the leaf node of the child node, and for the next rule of the rule, return to the step of inserting each field condition into the child node of the root node one by one.

[0133] According to an embodiment of the present disclosure, the field matching module 720 is also used to: for each key field, use a fuzzy matching algorithm to perform a first match between the leaf nodes of the multi-tree structure and the key field; when the results of the first matching of multiple key fields all fail, use a multi-field joint matching algorithm to perform a second match between the multi-tree structure and multiple key fields; when the results of the second matching fail, determine that the matching results of multiple key fields all fail.

[0134] According to an embodiment of the present disclosure, the field matching module 720 is also used to: for any target path from the root node to the leaf node of the multi-branch tree structure, match multiple key fields level by level according to the hierarchical structure of the multi-branch tree structure to obtain matching degrees at each level; assign a corresponding weighted score to each key field; perform weighted summation of the matching degrees at each level and the corresponding weighted scores to obtain the total matching degree between the multi-branch tree structure and the multiple key fields; determine whether the total matching degree is greater than a preset matching degree threshold, and if so, determine that the result of the second match is successful; otherwise, for another target path of the multi-branch tree structure, return to the step of matching multiple key fields level by level.

[0135] According to an embodiment of the present disclosure, the knowledge graph reasoning module 730 is also used to: configure multiple entities, entity attributes and entity relationships according to log data and error causes, wherein entities represent objects in source log data, and entity relationships represent associations between entities; for any two entities, calculate the first similarity between the corresponding two entity attributes; when the first similarity is greater than a set first similarity threshold, merge the two entities; for any two entity relationships, calculate the second similarity between the corresponding two relationship attributes; when the second similarity is greater than a set second similarity threshold, merge the two entity relationships.

[0136] According to an embodiment of the present disclosure, the knowledge graph reasoning module 730 is also used to preprocess the source log data and error causes before configuring multiple entities, entity attributes, and relationships between entities, wherein the preprocessing includes missing value processing, outlier cleaning, and duplicate data removal.

[0137] According to an embodiment of the present disclosure, the knowledge graph reasoning module 730 is also used to: use set reasoning rules to reason about log data, wherein the reasoning rules include entity matching rules and path search rules; and determine multiple associated paths of the log data based on the reasoning results.

[0138] According to an embodiment of the present disclosure, any multiple modules of the field extraction module 710, the field matching module 720, the knowledge graph reasoning module 730 and the error location module 740 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the field extraction module 710, the field matching module 720, the knowledge graph reasoning module 730 and the error location module 740 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the field extraction module 710, the field matching module 720, the knowledge graph reasoning module 730 and the error location module 740 can be at least partially implemented as a computer program module, which can perform corresponding functions when executed.

[0139] Figure 8 A block diagram of an electronic device suitable for implementing a method for processing log data according to an embodiment of the present disclosure is schematically shown.

[0140] like Figure 8As shown, the electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0141] In RAM 803, various programs and data required for the operation of electronic device 800 are stored. Processor 801, ROM 802 and RAM 803 are connected to each other via bus 804. Processor 801 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0142] According to an embodiment of the present disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the input / output (I / O) interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that the computer program read therefrom is installed into the storage portion 808 as needed.

[0143] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0144] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0145] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the log data processing method provided by the embodiment of the present disclosure.

[0146] The above functions defined in the system / device of the embodiment of the present disclosure are executed when the computer program is executed by the processor 801. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0147] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0148] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0149] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0150] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0151] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0152] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for processing log data, characterized in that: The method comprises: Obtaining log data with error information, and extracting multiple key fields of the log data; Obtain a preset rule set, and use a multi-branch tree structure constructed according to the rule set to match each of the key fields to obtain a matching result, wherein the matching result indicates whether the key field matches the error information; When the matching results of the multiple key fields all fail, using a pre-generated knowledge graph to infer multiple association paths of the log data; The multiple associated paths are screened based on the estimated path weights to obtain a propagation path of the log data to locate the error information.

2. The method according to claim 1, characterized in that The multi-branch tree structure constructed according to the rule set includes: Load the rule set from a specified path and determine the file type of the rule set; A parser corresponding to the file type is called to convert the rule set into a multi-branch tree structure.

3. The method according to claim 2, characterized in that The rule set includes a plurality of rules for identifying log data errors, each rule including at least one field condition, an error code and a rule description; Converting the rule set into a polytree structure includes: parsing the rule set into structured data; Initialize the root node of the multi-branch tree structure; Traversing each rule of the rule set of structured data, and inserting each field condition into the child node of the root node one by one according to whether each field condition of the rule exists in the multi-branch tree structure; When it is determined that the traversal of the rule is completed, the error code and rule description of the rule are stored in the leaf node of the child node, and for the next rule of the rule, the step of inserting each of the field conditions into the child node of the root node one by one is returned.

4. The method according to claim 1, characterized in that: The using of the multi-branch tree structure constructed according to the rule set to match each of the key fields to obtain a matching result includes: For each of the key fields, a fuzzy matching algorithm is used to perform a first match between the leaf nodes of the multi-branch tree structure and the key field; When the results of the first matching of the multiple key fields all fail, using a multi-field joint matching algorithm to perform a second matching between the multi-tree structure and the multiple key fields; When the result of the second matching fails, it is determined that the matching results of the multiple key fields all fail.

5. The method according to claim 4, characterized in that The adopting a multi-field joint matching algorithm to perform a second matching on the multi-tree structure and the multiple key fields includes: For any target path from the root node to the leaf node of the multi-branch tree structure, the multiple key fields are matched level by level according to the hierarchical structure of the multi-branch tree structure to obtain matching degrees at each level; Assigning a corresponding weighted score to each of the key fields; Performing weighted summation of the matching degrees at each level and the corresponding weighted scores to obtain a total matching degree between the multi-branch tree structure and the multiple key fields; Determine whether the total matching degree is greater than or equal to a preset matching degree threshold; if so, determine that the result of the second matching is successful; otherwise, for another target path of the multi-tree structure, return to the step of matching the multiple key fields level by level.

6. The method according to claim 1, characterized in that The knowledge graph is pre-generated in the following way: Obtain the source log data that has been processed for error reporting and its corresponding error reporting reasons; Integrate and optimize the source log data and the error cause to generate a knowledge graph, wherein integrating and optimizing the source log data and the error cause includes: According to the source log data and the error cause, configure multiple entities, entity attributes, and relationships between entities, wherein the entities represent objects in the source log data, and the relationships between entities represent associations between the entities; For any two entities, calculating a first similarity between the corresponding attributes of the two entities; When the first similarity is greater than a set first similarity threshold, merging the two entities; For any two of the entity relationships, calculate the second similarity between the corresponding two relationship attributes; When the second similarity is greater than a set second similarity threshold, the two inter-entity relationships are merged.

7. The method according to claim 6, characterized in that Before configuring the multiple entities, entity attributes, and relationships between entities, the following steps are also included: The source log data and the error cause are preprocessed, wherein the preprocessing includes missing value processing, outlier cleaning and duplicate data removal.

8. The method according to claim 1, characterized in that The using of the pre-generated knowledge graph to infer multiple association paths of the log data includes: Using the set inference rules to infer the log data, wherein the inference rules include entity matching rules and path search rules; Based on the inference results, multiple association paths of the log data are determined.

9. A log data processing device, characterized in that: The device comprises: A field extraction module, used to obtain log data marked with error information and extract multiple key fields of the log data; A field matching module, used to obtain a preset rule set, and use a multi-branch tree structure constructed according to the rule set to match each of the key fields to obtain a matching result, wherein the matching result indicates whether the key field matches the error information; A knowledge graph reasoning module, used for inferring multiple association paths of the log data using a pre-generated knowledge graph when the matching results of the multiple key fields all fail; The error locating module is used to screen the multiple associated paths based on the estimated path weights to obtain the propagation path of the log data so as to locate the error information.

10. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images