Log analysis method and device and log analysis architecture

Abstract

The invention relates to a log analysis method and device and a log analysis framework, and the method comprises the steps that an error log set is obtained, and error logs refer to log data generated when a node runs abnormally or fails; creating an error log template set based on the error log set; and matching a real-time log stream of a target node with an error log template in the error log template set, and generating a log analysis result of the target node based on a matching result. Therefore, the log analysis efficiency and the troubleshooting efficiency can be greatly improved.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular, to a log analysis method, apparatus, and log analysis architecture. Background Art

[0002] With the continuous expansion of the user scale of the data platform, the complexity and pressure of the operation and maintenance work have also increased sharply. During the operation and maintenance process, log data is an important clue for troubleshooting system failures.

[0003] Among them, log data is a set of files recording system operation events, which contains a large amount of information covering multiple aspects such as frameworks, services, and businesses. The information is complex and intertwined. This makes it often necessary for operation and maintenance personnel to "fish for a needle in a haystack" in the huge log data to find the root cause of the failure when a failure occurs. This process is not only time-consuming and laborious, but also extremely prone to omission or misjudgment, bringing great challenges to the operation and maintenance work. Summary of the Invention

[0004] This application provides a log analysis method, apparatus, and log analysis architecture to solve the technical problem that in the prior art, when a failure occurs, operation and maintenance personnel often need to "fish for a needle in a haystack" in the huge log data to find the root cause of the failure, which is time-consuming and laborious and extremely prone to omission or misjudgment.

[0005] In a first aspect, this application provides a log analysis method, and the method includes:

[0006] Obtain an error log set, where the error log refers to the log data generated when a node runs abnormally or fails;

[0007] Create an error log template set based on the error log set;

[0008] Match the real-time log stream of the target node with the error log templates in the error log template set, and generate a log analysis result of the target node based on the matching result.

[0009] In a possible implementation manner, the obtaining of the error log set includes:

[0010] According to the log level, filter out the error logs from the historical log stream of the target node to form an error log set corresponding to the target node.

[0011] In a possible implementation manner, the creating of the error log template set based on the error log set includes:

[0012] Preprocess the error logs in the error log set to obtain preprocessed error logs;

[0013] Convert the preprocessed error log into a vector representation;

[0014] Use a set clustering algorithm to cluster the vector representation of the preprocessed error log to obtain multiple clusters;

[0015] For each of the clusters, create a corresponding error log template according to the error log corresponding to the cluster to form an error log template set.

[0016] In a possible implementation manner, generating the log analysis result of the target node based on the matching result includes:

[0017] Based on the matching result, determine the number of times that the real-time log stream successfully matches the error log templates in the error log template set at each moment within a specific time period;

[0018] Based on the number of times corresponding to each moment of the real-time log stream within the specific time period, generate a visual log analysis chart of the real-time log stream within the specific time period, and use the visual log analysis chart as the log analysis result of the target node.

[0019] In a possible implementation manner, the method further includes:

[0020] When it is determined that the log in the real-time log stream fails to match any error log template in the error log template set, determine whether the log is an error log;

[0021] When it is determined that the log is an error log, create a new error log template according to the log and classify the new error log template into the error log template set.

[0022] In a second aspect, the present application provides a log analysis architecture, including:

[0023] A log collection middleware for obtaining log data generated by a log source;

[0024] A computing engine reads log data from the log collection middleware, obtains an error log set from the read log data, where the error log refers to log data generated when a node runs with an exception or a fault; creates an error log template set based on the error log set; and reads a real-time log stream of a target node from the log collection middleware, matches the real-time log stream of the target node with the error log templates in the error log template set, and based on the matching result, counts the number of times that the real-time log stream successfully matches the error log templates in the error log template set at each moment within a specific time period;

[0025] The display component obtains the above statistical results, generates a visual log analysis chart of the real-time log stream within the specific time period based on the statistical results, and displays the visual log analysis chart as the log analysis result of the target node.

[0026] In a possible implementation, the computing engine obtains an error log set from the read log data, including:

[0027] According to the log level, error logs from the target node are filtered out from the read log data to form an error log set corresponding to the target node.

[0028] In a possible implementation, the computing engine creates an error log template set based on the error log set, including:

[0029] Preprocess the error logs in the error log set to obtain preprocessed error logs;

[0030] Convert the preprocessed error logs into vector representations;

[0031] Use a set clustering algorithm to cluster the vector representations of the preprocessed error logs to obtain multiple clusters;

[0032] For each cluster, create a corresponding error log template according to the error logs corresponding to the cluster to form an error log template set.

[0033] In a possible implementation, the computing engine

[0034] In the case where it is determined that the log in the real-time log stream fails to match any error log template in the error log template set, determine whether the log is an error log;

[0035] In the case where it is determined that the log is an error log, create a new error log template according to the log and classify the new error log template into the error log template set.

[0036] In a third aspect, the present application provides a log analysis device, and the device includes:

[0037] A log collection module for obtaining an error log set, where the error log refers to log data generated when there is an abnormality or failure in the operation of the node;

[0038] A log template creation module for creating an error log template set based on the error log set;

[0039] A log analysis module for matching the real-time log stream of a target node with the error log templates in the error log template set and generating a log analysis result of the target node based on the matching result.

[0040] In a possible implementation manner, the log collection module is specifically configured to:

[0041] Filter out error logs from the historical log stream of the target node according to the log level to form an error log set corresponding to the target node.

[0042] In a possible implementation manner, the log template creation module includes:

[0043] A preprocessing unit for preprocessing the error logs in the error log set to obtain preprocessed error logs;

[0044] A vectorization unit for converting the preprocessed error logs into vector representations;

[0045] A clustering unit for clustering the vector representations of the preprocessed error logs by using a set clustering algorithm to obtain multiple clusters;

[0046] A creation unit for creating corresponding error log templates according to the error logs corresponding to each cluster to form an error log template set.

[0047] In a possible implementation manner, the log analysis module includes:

[0048] A statistics unit for determining, based on the matching result, the number of times the real-time log stream matches successfully with the error log templates in the error log template set at each moment within a specific time period;

[0049] A chart generation unit for generating a visual log analysis chart of the real-time log stream within the specific time period based on the number of times corresponding to each moment of the real-time log stream within the specific time period, and using the visual log analysis chart as the log analysis result of the target node.

[0050] In a possible implementation manner, the device further includes:

[0051] A judgment module for determining whether the log is an error log when it is determined that the log in the real-time log stream fails to match any of the error log templates in the error log template set;

[0052] A log template update module, which is used to create a new error log template according to the log when it is determined that the log is an error log, and classify the new error log template into the error log template set.

[0053] In a fourth aspect, the present application provides an electronic device, including: a processor and a memory, where the processor is configured to execute a log analysis program stored in the memory to implement the log analysis method according to any one of the first aspects.

[0054] In a fifth aspect, the present application provides a storage medium, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the log analysis method according to any one of the first aspects.

[0055] The above technical solutions provided by the embodiments of the present application have the following advantages compared with the prior art: In the method provided by the embodiments of the present application, by obtaining an error log set, creating an error log template set based on the error log set, matching the real-time log stream of the target node with the error log templates in the error log template set, and generating a log analysis result of the target node based on the matching result, it is possible to quickly find the log data that matches the error log template, that is, the error log, in a large amount of complex log data. This realizes the transformation from "log-oriented" to "category-oriented", enabling users to directly focus on the categories of various errors or warnings and their occurrence frequencies without having to review a large amount of log data item by item, thus greatly improving the log analysis efficiency and troubleshooting efficiency. Description of the Drawings

[0056] The accompanying drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0058] One or more embodiments are exemplarily illustrated by the pictures in the corresponding accompanying drawings. These exemplary illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements, unless otherwise stated, and the drawings in the drawings do not constitute a proportional limitation.

[0059] Figure 1 It is a flowchart of an embodiment of a log analysis method provided by an embodiment of the present application;

[0060] Figure 2 An example diagram of a visualization chart obtained by applying the method provided in the embodiments of the present application;

[0061] Figure 3 A flowchart of an embodiment of another log analysis method provided in the embodiments of the present application;

[0062] Figure 4 A schematic diagram of a log analysis architecture provided in the embodiments of the present application;

[0063] Figure 5 A block diagram of an embodiment of a log analysis device provided in the embodiments of the present application;

[0064] Figure 6 A schematic diagram of the structure of an electronic device provided in the embodiments of the present application. Detailed implementation manners

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some but not all of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application.

[0066] The following disclosure provides many different embodiments or examples for implementing different structures of the present application. To simplify the disclosure of the present application, components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the present application. In addition, the present application may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplification and clarity and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0067] To solve the technical problem in the prior art that when a fault occurs, operation and maintenance personnel often need to "fish for a needle in a haystack" in a large amount of log data to find the root cause of the fault, which is time-consuming and laborious, and is very likely to be omitted or misjudged, the present application provides a log analysis method, a device, and a log analysis architecture, which can greatly improve the log analysis efficiency and troubleshooting efficiency.

[0068] Figure 1 A flowchart of an embodiment of a log analysis method provided in the embodiments of the present application.

[0069] As Figure 1 shown, the method includes the following steps:

[0070] Step 101: Obtain an error log set. Here, an error log refers to the abnormal data generated when a node runs with an exception or a fault.

[0071] In the embodiments of the present application, a node is a general term for a system, an application, a server, a process, or a service, etc. The specific reference to a node is not limited herein. An error log refers to the log data generated when a node runs with an exception or a fault. Such logs usually contain detailed error information, exception types, occurrence times, components or services that may be affected, and possible causes of the errors, etc.

[0072] In one embodiment, an error log set collects error logs from multiple log sources (such as nodes). Correspondingly, in step 101, obtaining the error log set includes: collecting log data from the historical log streams of multiple log sources, then screening out the error logs from the collected log data, and classifying the screened error logs into the same log set to form an error log set.

[0073] In another embodiment, an error log set collects error logs from one log source, such as a target node. Here, the target node refers to the node to be analyzed for logs. Correspondingly, in step 101, obtaining the error log set includes: collecting log data from the historical log stream of the target node, then screening out the error logs from the collected log data, and classifying the screened error logs into the same log set to form an error log set corresponding to the target node.

[0074] In addition, in this embodiment, for different nodes, the error log sets corresponding to the respective different nodes can be obtained separately. Subsequently, when analyzing the logs of a certain node, the error log set corresponding to that node is used and the log analysis method provided by the embodiments of the present application is applied to analyze the logs of that node.

[0075] In the above embodiments, as an optional implementation manner, when screening error logs from the original log data, the screening of error logs can be implemented according to the log level. For example, the specific implementation of obtaining the error log set in step 101 includes: screening out error logs from the historical log stream of the target node according to the log level to form an error log set corresponding to the target node.

[0076] Among them, the log levels of error logs may include Error, Warning, Critical, etc., which specifically depend on the configuration and classification criteria of the log system, and the embodiments of the present application do not limit this. For example, the following is an example of a piece of original log data:

[0077] “2024-11-06 09:13:19 Error org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: NotServingRegionException: 1 time, servers with issues: node71-66-99-bdxs.qiyi.hadoop, 60020, 1704803004265”

[0078] If the log level of the above original log data is “Error”, then in step 101, this log data can be determined as an error log and classified into the error log set.

[0079] Step 102: Create an error log template set based on the error log set.

[0080] The above error log template set may include one or more error log templates. The so-called error log template refers to: a general log template designed for a specific type of error situation, which includes the common features and patterns of different log data for the same error situation. Thus, an error log template represents a specific type of error or warning message and can be used to match and record specific types of errors or warnings. For example, the database connection error template is used to record the situation of database connection failure, including fields such as error messages, the database URL attempted to connect, and the number of retries, which helps diagnose network problems, authentication failures, or configuration errors.

[0081] In one embodiment, an error log set collects error logs from multiple log sources. Then, in step 102, the error log template created based on the error log set will have wide applicability, be able to cover various different types of errors and events, and can be used for cross-log source error analysis, monitoring, and alarm, helping operation and maintenance personnel quickly locate and solve problems.

[0082] In another embodiment, an error log set collects error logs from one log source. Then, in step 102, the error log template set created based on the error log set will be targeted. This targeting is reflected in that the error log template will closely conform to the log format and content of this log source, so as to be able to specifically cover the errors and events that may occur on this log source and can be used to assist in more accurately matching and identifying the error logs in this log source. This will help operation and maintenance personnel more efficiently and accurately monitor and analyze a specific log source, such as the running status of the target node, and discover and solve problems in a timely manner.

[0083] Correspondingly, in this embodiment, for different nodes, the error log sets corresponding to the different nodes can be respectively obtained, and subsequently, for each node, an error log template corresponding to the node is created based on the error log set corresponding to the node. Then, when performing log analysis on a specific node, such as a target node, the error log template corresponding to the target node is used to perform log analysis on the log stream of the target node.

[0084] Step 103: Match the real-time log stream of the target node with the error log templates in the error log template set, and generate a log analysis result of the target node based on the matching result.

[0085] Here, the purpose of matching the real-time log stream of the target node with the error log templates in the error log template set is to identify the log data in the real-time log stream of the target node that matches the error log templates. These log data can be used to describe the error or warning events that occur during the operation of the target node.

[0086] In one embodiment, the specific implementation of generating a log analysis result of the target node based on the matching result between the real-time log stream of the target node and the error log templates in the error log template set includes: based on the matching result, determining the number of times of successful matching with the error log templates in the error log template set at each moment within a specific time period of the real-time log stream; generating a visual log analysis chart of the real-time log stream of the target node within the specific time period based on the number of times corresponding to each moment within the specific time period of the real-time log stream, and using the visual log analysis chart as the log analysis result of the target node.

[0087] Among them, as an optional implementation manner, for the real-time log stream of the target node within a specific time period, the log data at each moment can be matched with the templates in the error log template set, and the number of times of successful matching can be counted. Subsequently, these statistical results are visually displayed in the form of a time-times chart. In this chart, the horizontal axis represents time, and the vertical axis represents the number of times of successful matching. This chart can intuitively display the number of logs that successfully match the error log templates at each time node. Through this intuitive chart display, users can clearly understand the matching situation between the log stream of the target node and the error log template set, so as to more effectively perform log analysis and troubleshooting.

[0088] As another alternative implementation, based on the above implementation, it is also possible to further distinguish the number of times different error log templates match successfully. That is to say, in addition to counting the total number of successful matches according to time nodes, the number of times each error log template matches successfully at the corresponding time node is specifically recorded. This enables the visualization chart to not only clearly reflect the changing trend of error or warning events in the target node log stream over time, but also accurately distinguish the distribution of different types of error or warning events. In this visualization chart, the horizontal axis represents time, and the vertical axis uses a grouped form to display the number of times different error log templates match successfully. For example, referring to Figure 2 for an example, each error log template will have a dot to represent the number of times it matches successfully, and the dots of different error log templates at the same moment are distinguished by color or label.

[0089] This processing method enables users to more accurately identify the changing trend and distribution of different types of error or warning events in the system over time, thereby helping users take more targeted measures to solve problems and optimize system performance.

[0090] The technical solution provided by the embodiments of the present application, by obtaining an error log set, creating an error log template set based on the error log set, matching the real-time log stream of the target node with the error log templates in the error log template set, and generating a log analysis result of the target node based on the matching result, can quickly find the log data that matches the error log template, that is, the error log, in a large amount of complex log data. This realizes the transformation from "log-oriented" to "category-oriented", enabling users to directly focus on the categories of various errors or warnings and their occurrence frequencies without having to review a large amount of log data item by item, thus greatly improving the log analysis efficiency and troubleshooting efficiency.

[0091] Figure 3 It is a flowchart of an embodiment of another log analysis method provided by the embodiments of the present application. Figure 3 The process shown is Figure 1 Based on the process shown, it describes how to create an error log template set based on an error log set. As Figure 3 shown, it includes the following steps:

[0092] Step 301: Preprocess the error logs in the error log set to obtain preprocessed error logs.

[0093] The above preprocessing may include information filtering and formatting operations, so that the preprocessed error logs have a specific data structure, thus facilitating subsequent data processing processes. The following is an example of preprocessed error logs:

[0094]

[0095]

[0096] In the above example, jobName represents the task name, level represents the log level, logdir represents the directory where the logs are located, loggerName represents the log name, message represents the log content, projectId represents the ID of the project, service represents the service provider, thread represents the thread name, and timeMillis represents the timestamp.

[0097] Step 302: Convert the preprocessed error logs into vector representations.

[0098] In step 302, the preprocessed error logs are vectorized. Vectorization is the process of converting text data into a numerical form, which typically involves using techniques such as the bag-of-words model, TF-IDF (term frequency-inverse document frequency), word embeddings (such as Word2Vec, BERT, etc.). Through these methods, each error log is converted into a high-dimensional vector, where each dimension of the vector represents the presence or importance of a certain feature (such as a word) in the log.

[0099] Step 303: Use the set clustering algorithm to cluster the vector representations of the preprocessed error logs to obtain multiple clusters.

[0100] Step 303 aims to divide the error logs into multiple clusters, and the logs within each cluster have similar features or patterns, thereby characterizing different types of errors or warnings.

[0101] In one embodiment, the set clustering algorithm is the native Drain3 algorithm. The Drain3 algorithm is a clustering method designed specifically for log data and has powerful log parsing and clustering capabilities. It can accurately identify and extract key information in the logs, such as timestamps, log levels, source components, and specific error messages or warning contents. Based on this information, the Drain3 algorithm can effectively measure the similarity between log entries in the high-dimensional vector space and merge similar log entries into the same cluster. Compared with other clustering algorithms, Drain3 shows higher efficiency and accuracy when processing log data. Therefore, choosing Drain3 as the clustering algorithm here can quickly and accurately mine valuable information from the preprocessed error logs and create more representative error log templates for each cluster.

[0102] Step 304: For each cluster, create a corresponding error log template according to the error logs corresponding to the cluster to form an error log template set.

[0103] In step 304, corresponding error log templates are created for each cluster. The process of creating templates usually involves extracting the common parts or features of the logs in the cluster to form a concise and representative template. For example, if the logs in a certain cluster all contain the key information "service crash caused by insufficient memory", then this key information can be extracted as an error log template. Through this process, one or more error log templates can be generated for each cluster, and finally an error log template set containing multiple templates is formed.

[0104] Figure 3 The process shown realizes the creation of an error log template set from the preprocessed error logs, which provides an implementation basis for subsequent log analysis tasks such as fast and accurate log matching and fault troubleshooting.

[0105] Based on any of the above embodiments, the present application also provides the following embodiment:

[0106] In the case where it is determined that the log in the real-time log stream of the target node fails to match any of the error log templates in the error log template set, determine whether the log is an error log; in the case where it is determined that the log is an error log, create a new error log template according to the log and classify the new error log template into the error log template set.

[0107] In the above embodiment, when processing the real-time log stream of the target node, if a certain log fails to match all the templates in the error log template set and this log is an error log, it means that a new type of error appears. In this regard, a new error log template is created according to this log and classified into the error log template set for subsequent matching and identification. Applying the above embodiment ensures that the system can continuously learn and adapt to new error types and maintain the effectiveness and accuracy of its log analysis.

[0108] Figure 4 It is a schematic diagram of a log analysis architecture provided by an embodiment of the present application. As Figure 4 shown, the architecture includes:

[0109] A log collection middleware 41 that obtains log data generated by a log source;

[0110] The computing engine 42 reads log data from the log collection middleware, obtains an error log set from the read log data, where the error log refers to the log data generated when there are exceptions or faults in the node operation; creates an error log template set based on the error log set; and reads the real-time log stream of the target node from the log collection middleware, matches the real-time log stream of the target node with the error log templates in the error log template set, and based on the matching result, counts the number of times the real-time log stream successfully matches the error log templates in the error log template set at each moment within a specific time period.

[0111] The display component 43 obtains the above statistical result, generates a visual log analysis chart of the real-time log stream within the specific time period based on the statistical result, and displays the visual log analysis chart as the log analysis result of the target node.

[0112] In a possible implementation manner, the log collection middleware 41 adopts a Kafak message middleware. The Kafak message middleware accesses the log source and can obtain and cache the log data generated by the log source.

[0113] The computing engine 42 adopts a Flink framework. The Flink framework provides a UDF (User-defined Function, extended development mechanism). The UDF can be used to encapsulate algorithms in a way that calls frequently used or custom logic that is difficult to express in other ways in a query statement, and register the algorithm as a function for calling. Based on this, the original Drain3 algorithm code is integrated into the Flink framework to form the computing engine 42 capable of implementing the technical solution of the embodiment of the present application.

[0114] Among them, the Drain3 algorithm provides two most core functions:

[0115] 1) template_miner.match(message): used to match the log with the log template. If the corresponding log template is matched, the template ID is returned;

[0116] 2) template_miner.add(message): used to insert an original log, and returns the template ID if the insertion is successful.

[0117] In the computing engine 42, a model training task is developed using Flink SQL. Model training means passing the log content into the function template_miner.add(log) registered as a UDF. The template_miner object can be understood as a continuously trained log template tree. All log templates are recorded in this tree and will be saved periodically to avoid loss. This continuous insertion process is called model training.

[0118] In the computing engine 42, an aggregation task is developed. The aggregation task refers to aggregating the log stream to be analyzed according to a certain time window and applying the technical solution provided by the embodiments of the present application for aggregation statistics to obtain an aggregation result.

[0119] The display component 43 adopts Paimon (a display component in lake format). This display component can generate a visualization chart from the above aggregation result with the help of a visualization tool and display it.

[0120] In a possible implementation, the computing engine 42 obtains an error log set from the read log data, including: screening out error logs from the target node from the read log data according to the log level to form an error log set corresponding to the target node.

[0121] Among them, the computing engine 42 can subscribe to the log collection middleware 41. Then, when the log data is written into the log collection middleware 41, the computing engine 42 can read the log data from the log collection middleware 41 based on the subscription mechanism.

[0122] In a possible implementation, the computing engine 42 creates an error log template set based on the error log set, including: preprocessing the error logs in the error log set to obtain preprocessed error logs; converting the preprocessed error logs into vector representations; using a set clustering algorithm to cluster the vector representations of the preprocessed error logs to obtain multiple clusters; for each cluster, creating a corresponding error log template according to the error logs corresponding to the cluster to form an error log template set.

[0123] In a possible implementation, when the computing engine 42 determines that the log in the real-time log stream fails to match any error log template in the error log template set, it determines whether the log is an error log; if it determines that the log is an error log, it creates a new error log template according to the log and incorporates the new error log template into the error log template set.

[0124] Figure 4 For the detailed working process of the shown system architecture, reference can be made to the description in the above method embodiments, which will not be elaborated here.

[0125] Figure 5 This is a block diagram of an embodiment of a log analysis device provided by an embodiment of the present application. As Figure 5 shown, the device includes:

[0126] A log collection module 51, configured to obtain an error log set, where the error log refers to log data generated when there is an abnormality or failure in the operation of a node;

[0127] A log template creation module 52, configured to create an error log template set based on the error log set;

[0128] A log analysis module 53, configured to perform matching processing on the real-time log stream of a target node and the error log templates in the error log template set, and generate a log analysis result of the target node based on the matching result.

[0129] In a possible implementation manner, the log collection module 51 is specifically configured to:

[0130] According to the log level, filter out error logs from the historical log stream of the target node to form an error log set corresponding to the target node.

[0131] In a possible implementation manner, the log template creation module 52 includes:

[0132] A preprocessing unit, configured to preprocess the error logs in the error log set to obtain preprocessed error logs;

[0133] A vectorization unit, configured to convert the preprocessed error logs into vector representations;

[0134] A clustering unit, configured to perform clustering processing on the vector representations of the preprocessed error logs by using a set clustering algorithm to obtain multiple clusters;

[0135] A creation unit, configured to, for each of the clusters, create a corresponding error log template according to the error logs corresponding to the cluster to form an error log template set.

[0136] In a possible implementation manner, the log analysis module 53 includes:

[0137] A statistics unit, configured to, based on the matching result, determine the number of times of successful matching between the real-time log stream and the error log templates in the error log template set at each moment within a specific time period;

[0138] A chart generation unit, configured to generate a visual log analysis chart of the real-time log stream within the specific time period based on the corresponding number of times at each moment of the real-time log stream within the specific time period, and use the visual log analysis chart as the log analysis result of the target node.

[0139] In a possible implementation manner, the device further includes:

[0140] A judgment module, configured to determine whether the log is an error log when it is determined that the log in the real-time log stream fails to match any error log template in the error log template set;

[0141] A log template update module, configured to create a new error log template according to the log and classify the new error log template into the error log template set when it is determined that the log is an error log.

[0142] As Figure 6 shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114. Among them, the processor 111, the communication interface 112, and the memory 113 complete mutual communication through the communication bus 114.

[0143] The memory 113 is used to store a computer program;

[0144] In an embodiment of the present application, when the processor 111 is used to execute the program stored on the memory 113, it implements the log analysis method provided by any one of the foregoing method embodiments, including:

[0145] Obtain an error log set, where the error log refers to log data generated when there is an abnormality or failure in the operation of the node;

[0146] Create an error log template set based on the error log set;

[0147] Match the real-time log stream of the target node with the error log templates in the error log template set, and generate a log analysis result of the target node based on the matching result.

[0148] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the log analysis method provided by any one of the foregoing method embodiments.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0151] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" as used herein may also include the plural forms. The terms "include", "comprise", "contain", and "have" are inclusive and thus specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or their combinations. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be executed in the particular order described or illustrated, unless the execution order is explicitly stated. It should also be understood that additional or alternative steps can be used.

[0152] The above description is only the specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A log analysis method, characterized in that: The method comprises: Get the error log set, where the error log refers to the log data generated when the node runs abnormally or fails; Creating an error log template set based on the error log set; The real-time log stream of the target node is matched with the error log template in the error log template set, and the log analysis result of the target node is generated based on the matching result.

2. The method according to claim 1, characterized in that The obtaining of the error log set includes: According to the log level, the error log is screened out from the historical log stream of the target node to form an error log set corresponding to the target node.

3. The method according to claim 1, characterized in that The step of creating an error log template set based on the error log set includes: Preprocessing the error logs in the error log set to obtain preprocessed error logs; Convert the preprocessed error log into a vector representation; Using a set clustering algorithm to perform clustering processing on the vector representation of the preprocessed error log to obtain multiple clusters; For each of the clusters, a corresponding error log template is created according to the error log corresponding to the cluster to form an error log template set.

4. The method according to claim 1, characterized in that The generating the log analysis result of the target node based on the matching result includes: Based on the matching result, determining the number of times that the real-time log stream successfully matches the error log template in the error log template set at each moment in a specific time period; Based on the number of times the real-time log stream corresponds to each moment in the specific time period, a visual log analysis chart of the real-time log stream in the specific time period is generated, and the visual log analysis chart is used as the log analysis result of the target node.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In the case where it is determined that the log in the real-time log stream fails to match any error log template in the error log template set, determining whether the log is an error log; In the case where it is determined that the log is an error log, a new error log template is created according to the log, and the new error log template is included in the error log template set.

6. A log analysis architecture, characterized in that: include: Log collection middleware, which obtains log data generated by log sources; A computing engine, which reads log data from the log collection middleware, obtains an error log set from the read log data, wherein the error log refers to log data generated when an abnormality or failure occurs in the operation of a node; creates an error log template set based on the error log set; and reads the real-time log stream of the target node from the log collection middleware, matches the real-time log stream of the target node with the error log template in the error log template set, and based on the matching result, counts the number of times the real-time log stream successfully matches the error log template in the error log template set at each moment in a specific time period; A display component obtains the above statistical results, generates a visual log analysis chart of the real-time log stream within the specific time period based on the statistical results, and displays the visual log analysis chart as the log analysis result of the target node.

7. A log analysis device, characterized in that: The device comprises: The log collection module is used to obtain an error log set, where the error log refers to the log data generated when there is an abnormality or failure in the node operation; A log template creation module, used to create an error log template set based on the error log set; The log analysis module is used to match the real-time log stream of the target node with the error log template in the error log template set, and generate the log analysis result of the target node based on the matching result.

8. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a log analysis program stored in the memory to implement the log analysis method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the log analysis method according to any one of claims 1 to 6.

Images