Log analysis method, device and equipment and computer readable storage medium

Abstract

The application discloses a log analysis method and device, equipment and a computer readable storage medium, and relates to the technical field of information processing, to solve the problem of low automation degree of the log analysis method. The method comprises the following steps: acquiring feature information of each log in a log data set to be analyzed; constructing a hierarchical tree model corresponding to the feature information based on a minimum spanning tree; performing clustering processing on the logs in the log data set to be analyzed based on the hierarchical tree model, and acquiring a tree structure corresponding to each clustering type according to a clustering result; performing a spanning tree processing on the logs of a target clustering type according to the target clustering type and a target tree structure corresponding to the target clustering type, to obtain an analysis template spanning tree of the target clustering type; and performing analysis on the logs of the target clustering type according to the analysis template spanning tree, to obtain a log analysis result. The embodiment of the application can realize the automation of the whole process of log analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, and in particular to a log analysis method, device, equipment and computer readable storage medium. BACKGROUND

[0002] In the related art, the conventional process of log analysis by ELK usually needs to manually write an analysis file according to log content, which is composed of multiple modules, and each module is specially designed for log analysis of different formats or types. This method can almost handle all types of log analysis requirements, but its significant defect is that it needs manual writing of analysis files and configuration of regular expressions for log analysis. Some technical solutions analyze logs by pre-setting an analysis rule library, wherein the analysis rule library still needs manual intervention.

[0003] Therefore, the log analysis method in the related art has the defect of low automation degree. SUMMARY

[0004] Embodiments of the present application provide a log analysis method, device, equipment and computer readable storage medium, which can solve the technical problem of low automation degree of log analysis method.

[0005] In a first aspect, the embodiments of the present application provide a log analysis method, comprising:

[0006] obtaining feature information of each log in a log data set to be analyzed;

[0007] constructing a hierarchical tree model corresponding to the feature information based on a minimum spanning tree;

[0008] performing clustering processing on logs in the log data set to be analyzed based on the hierarchical tree model, and obtaining a tree structure corresponding to each clustering type in the clustering result according to the clustering result;

[0009] performing spanning tree processing on logs of a target clustering type according to the target clustering type and a target tree structure corresponding to the target clustering type, to obtain an analysis template spanning tree of the target clustering type, wherein the clustering types in the clustering result include the target clustering type;

[0010] performing analysis on logs of the target clustering type according to the analysis template spanning tree to obtain a log analysis result.

[0011] In a second aspect, the embodiments of the present application further provide a log analysis device, comprising:

[0012] a first obtaining module configured to obtain feature information of each log in a log data set to be analyzed;

[0013] A model construction module is configured to construct a hierarchical tree model corresponding to the feature information based on a minimum spanning tree.

[0014] A clustering module is configured to perform clustering processing on logs in the log data set to be parsed based on the hierarchical tree model, and obtain a tree structure corresponding to each clustering type in the clustering result.

[0015] A spanning tree module is configured to perform spanning tree processing on logs of a target clustering type according to the target clustering type and a target tree structure corresponding to the target clustering type, to obtain a parsing template spanning tree of the target clustering type, wherein the clustering types in the clustering result include the target clustering type.

[0016] A log parsing module is configured to parse logs of the target clustering type according to the parsing template spanning tree, to obtain a log parsing result.

[0017] In a third aspect, an embodiment of the present application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the log parsing method when executing the computer program.

[0018] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the steps in the log parsing method.

[0019] In a fifth aspect, an embodiment of the present application further provides a computer program product, which comprises computer instructions, and the computer instructions are executable on a processor to implement the steps in the log parsing method.

[0020] In this embodiment, feature information of each log entry in the log dataset to be parsed is obtained; a hierarchical tree model corresponding to the feature information is constructed based on a minimum spanning tree; the logs in the log dataset to be parsed are clustered based on the hierarchical tree model, and the tree structure corresponding to each cluster type in the clustering results is obtained; the logs of the target cluster type are processed by spanning tree according to the target cluster type and the target tree structure corresponding to the target cluster type to obtain the parsing template spanning tree of the target cluster type, wherein the cluster types in the clustering results include the target cluster type; the logs of the target cluster type are parsed according to the parsing template spanning tree to obtain the log parsing result. After obtaining the log feature information, a hierarchical tree model corresponding to the feature information is constructed based on a minimum spanning tree, and the logs in the log dataset to be parsed are clustered based on the hierarchical tree model. Finally, based on spanning tree technology, a parsing template spanning tree is automatically generated for each cluster type of log as the parsing rule applicable to that cluster type of log, so as to achieve efficient parsing of various types of logs. This method can automate the entire log parsing process and significantly improve log parsing efficiency. Among them, the clustering process is optimized based on the idea of ​​hierarchical clustering, so that the hierarchical tree model can automatically obtain the optimal clustering result only by giving the minimum sample contained in the cluster, avoiding the process of manual parameter tuning. In addition to improving the automation of the log parsing process, the accuracy and applicability of the hierarchical tree model can also be improved. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is one of the flowcharts of the log parsing method provided in the embodiments of this application;

[0023] Figure 2 This is a schematic diagram of the target tree structure update process provided in the embodiments of this application;

[0024] Figure 3 This is the second flowchart of the log parsing method provided in the embodiments of this application;

[0025] Figure 4 This is a structural diagram of the log parsing device provided in the embodiments of this application;

[0026] Figure 5 This is a structural diagram of the electronic device provided in the embodiments of this application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort are within the protection scope of the present application.

[0028] The log analysis method, log analysis device, electronic device, computer readable storage medium and computer program product provided by the embodiments of the present application will be described in detail below in conjunction with the drawings and specific embodiments and application scenarios.

[0029] Referring to Figure 1 , Figure 1 is a flowchart of the log analysis method provided by the embodiments of the present application, as shown in Figure 1 , comprising the following steps:

[0030] Step 101, obtaining feature information of each log in a to-be-analyzed log data set.

[0031] In some embodiments, the logs in the to-be-analyzed log data set can include multiple types of logs, such as system logs, application program logs, security logs, etc.

[0032] In some embodiments, the logs in the to-be-analyzed log data set can include logs in a diversified computing environment. The diversified computing environment includes but is not limited to various host and server systems, and operating systems, application programs and multiple software components running in these systems. A large amount of log information is generated in the diversified computing environment, and the nature, application range and scale of these log data present diversity. The log files contain rich information resources, which have high value for users. By analyzing different types of logs, users can extract diversified intelligence.

[0033] For ease of illustration, the following email service log is usually taken as an example for illustration in the embodiments of the present application:

[0034] Oct 6 00:00:13 localhost postfix / smtpd

[26072] :

[0035] disconnect from n169-113.mail.139.com[120.232.169.113] ehlo=1 mail=1rcpt=9 / 10 data=1 quit=1 commands=13 / 14.

[0036] By parsing the above email service log, key information can be extracted, including the timestamp of the event occurrence, the client identification initiating the connection, the Internet Protocol (IP) address of the client, and the statistical data of the related command execution. However, the unprocessed log data often has the problem of poor readability, which limits the in-depth analysis and statistical research of its content. Therefore, in order to effectively utilize the log information, it must be parsed.

[0037] In some embodiments, the collection strategy of the log data set can be mainly divided into local collection and remote collection according to the difference of applications or components. Local collection involves extracting data directly from the log storage location using standard tools under the condition of knowing the log file storage path and having the corresponding permissions. Remote collection requires the target of collection to configure the log tool, specify the local log source and the sending target (i.e. the central log server), and configure on the central log server to complete the reception of the log. The remote collection process includes installing and configuring the log collection tool on the log source device, establishing a secure transmission mechanism, and setting the receiving rules on the central server to ensure the effective transmission and storage of log data.

[0038] In some embodiments, after collecting the original log data, the original log data can be preprocessed to obtain the log data set to be parsed.

[0039] Among them, the preprocessing of the original log data can include at least one of the following:

[0040] Filtering invalid data, based on the fact that there may be a large amount of invalid data in the original log data, such as error records, duplicate records, blank lines, comments, duplicate log entries, etc., by cleaning the original log data, the efficiency of the logs in the log data set to be parsed can be improved;

[0041] Format unification, the format of the log is unified, the original log data comes from various sources, and the format of the log of different sources needs to be unified, such as converting the log into json or xml format;

[0042] Outlier processing, the log data can be detected by an outlier detection tool to find outliers and change or delete the outliers;

[0043] Null value processing, missing null values in the log can be identified and the null values can be filled or deleted.

[0044] In some embodiments, for each log in the log data set to be parsed, a feature extraction algorithm, an artificial intelligence (AI) model, a machine learning model, or the like can be used to extract features from the log and obtain feature information.

[0045] In some other embodiments, for each log in the log data set to be parsed, the log can be divided into a plurality of segmented words, and the feature information of the log can be determined according to the word frequency and the inverse document frequency index of the segmented words.

[0046] As an optional embodiment, the feature information of each log in the log data set to be parsed is obtained by:

[0047] performing feature extraction processing on each log in the log data set to be parsed to obtain the feature information of each log;

[0048] The feature extraction processing includes:

[0049] performing first segmentation processing on the log based on a regular segmenter or a dictionary to obtain U segmented words of the log, U being an integer greater than 1;

[0050] determining a first feature vector based on the feature values of the U segmented words, the first feature vector being a U-dimensional vector;

[0051] performing dimension reduction processing on the first feature vector to obtain a second feature vector, the feature information of each log including the second feature vector, the second feature vector being a two-dimensional feature vector.

[0052] In some embodiments, the first segmentation processing on the log based on the regular segmenter or the dictionary can be performed on each segmented word in the log, and the segmented words can be segmented words having linguistic meaning, i.e., the segmented words can not include character segments consisting of only symbols and numbers.

[0053] For example, for the following email service log:

[0054] Oct 6 00:00:13 localhost postfix / smtpd

[26072] :

[0055] disconnect from n169-113.mail.139.com[120.232.169.113] ehlo=1 mail=1rcpt=9 / 10 data=1 quit=1 commands=13 / 14.

[0056] After the first tokenization processing, the U segment words obtained are: [‘localhost’, ‘postfixsmtpd’, ‘disconnect’, ‘from’, ‘n169-113.mail.139.com’, ‘ehlo’, ‘mail’, ‘rcpt’, ‘data’, ‘quit’, ‘commands’], and in this embodiment, U equals 11.

[0057] Then, the feature value of each segment word is calculated.

[0058] For example, the term frequency-inverse document frequency (TF-IDF) of each segment word can be calculated using the following formula:

[0059] tf-idf(t,d)=tf(t,d)×idf(t);

[0060] wherein tf(t,d) represents the term frequency of the segment word t in the document d, such as tf(t,d)=the number of occurrences of the segment word t in the document d / the total number of segment words in the document d, and the document d can be the document composed of all segment words in the current log; idf(t) is the inverse document frequency of the segment word t, used to measure the rarity of the segment word t, such as idf(t)=log(N / df(t)), N represents the number of logs in the log data set to be parsed, and df(t) represents the number of logs containing the segment word t.

[0061] For example, assuming that each log in the log data set to be parsed has the segment word localhost, the TF-IDF feature value of the segment word localhost in the log is tf-idf=1 / 11*log(1)=0.

[0062] It is worth mentioning that, considering that the TF-IDF is simple to implement, has low computational cost, and can meet the demand that the log content contains less semantic information, using the TF-IDF value as the feature value of the segment word can improve the efficiency of extracting the first feature information.

[0063] In this embodiment, the first feature vector can be obtained by combining the feature values of U segmented words, and the dimension of the first feature vector is equal to U. Since the dimension of the text feature vector obtained by most text feature vector extraction methods is generally high, such as 10, 20 or even higher, the dimension of the first feature vector is high, and high-dimensional data is prone to poor distinguishability, which leads to overfitting in clustering analysis. By performing dimension reduction processing on the first feature vector, the dimension of the second feature vector obtained can be reduced to two dimensions. In this way, subsequent processing based on the second feature vector, such as minimum spanning tree processing to construct a hierarchical tree model and clustering of logs in the log data set to be analyzed based on the hierarchical tree model, can reduce overfitting and improve the efficiency of these processes.

[0064] In some embodiments, the dimension reduction processing on the first feature vector can be performed using the gradient descent method to find the optimal solution of the second feature vector.

[0065] In some embodiments, the t-distributed stochastic neighbor embedding (t-SNE) algorithm can be used to perform dimension reduction processing on the first feature vector. This algorithm uses a symmetric loss function when using the gradient descent method to find the optimal solution of the second feature vector, and uses a t-distribution to construct data distribution in low-dimensional space to solve the data congestion problem.

[0066] For example, the following parameters can be input to the t-SNE algorithm: N first feature vectors {X1, X2, X3,..., Xn} of U dimensions, the number of iterations T, the learning efficiency β, and the momentum a(t); then, N second feature vectors of two dimensions output by the t-SNE algorithm can be obtained.

[0067] Wherein, the logic process of reducing the first feature vector of U dimensions to the second feature vector of two dimensions based on the t-SNE algorithm is similar to the process of dimension reduction processing based on the t-SNE algorithm in related technologies.

[0068] For example, first calculate the conditional probability P j|i in high-dimensional space, then use a normal distribution N(0,10 -4 to randomly initialize Y, iterate from 1 to T rounds, and calculate the conditional probability q ij in low-dimensional space, the gradient of the loss function C(y i ) with respect to y i , and update the value of Y accordingly, so that the similarity (P j|i distribution) in high-dimensional space and the similarity (q ijThe relative entropy (Kullback-Leibler, KL) divergence is minimized by gradient descent to ensure that the low-dimensional second feature vectors preserve the structure of the high-dimensional first feature vectors as much as possible.

[0069] In step 102, a hierarchical tree model corresponding to the feature information is constructed based on the minimum spanning tree.

[0070] In some embodiments, taking a density-based clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) with noise as an example, the clustering process of the DBSCAN algorithm does not depend on the number of clusters and can automatically discover the cluster structure in the data; and is not sensitive to outliers and noise. However, the two parameters of the minimum step (Minpts) and the field radius (Eps) in the optimization process need to be manually adjusted constantly along with the clustering process.

[0071] In the present embodiment, the DBSCAN algorithm can be improved based on the idea of hierarchical clustering, and a minimum spanning tree is used to construct a hierarchical tree model between the second feature vectors, so that the model can automatically obtain the optimal clustering result only by giving the minimum samples contained in the cluster, avoiding the complex parameter adjustment process, and the accuracy and scope of application of the hierarchical tree model are improved.

[0072] In step 103, the logs in the to-be-analyzed log data set are clustered based on the hierarchical tree model, and the tree structure corresponding to each clustering type in the clustering result is obtained according to the clustering result.

[0073] In some embodiments, the second feature vectors of all logs in the to-be-analyzed log data set can be clustered based on the DBSCAN algorithm improved based on the idea of hierarchical clustering in step 102, so that the efficiency of the clustering process can be improved based on the low-dimensional characteristics of the second feature vectors, and the DBSCAN algorithm improved based on the idea of hierarchical clustering makes the model automatically obtain the optimal clustering result only by giving the minimum samples contained in the cluster, avoiding the complex parameter adjustment process.

[0074] Of course, in other embodiments, the feature information of the log can also be other vectors or other forms of feature information other than vectors, and at this time, a suitable clustering method can be selected according to the specific type and size of the feature information, which is not limited here.

[0075] Step 104, according to the target clustering type and the target tree structure corresponding to the target clustering type, performing a spanning tree processing on the logs of the target clustering type to obtain a parsing template spanning tree of the target clustering type, wherein the clustering types in the clustering result include the target clustering type.

[0076] In some embodiments, after obtaining the clustering result of the logs in the log data set to be parsed, the logs can be divided into log types corresponding to the clustering result, and each log type can correspond to a respective tree structure, wherein the tree structure can represent the log parsing template of the corresponding log type.

[0077] In some embodiments, the tree structure can be related to the size, structural distribution characteristics, and belonging log type of the logs of the corresponding log type.

[0078] For example, the logs of the same log type have the same number of segmented words, L, which is an integer greater than 1. At this time, the initialized tree structure can include L+1 nodes, wherein the first node is the root node, the following L nodes correspond to the L segmented words in the log, and finally, it is ended with a special symbol “$”. That is, a path from the root node P to “$” in the tree structure corresponds to the segmented word sequence in a log.

[0079] The parsing template spanning tree can represent the parsing rule of the logs of this type, and based on the parsing template spanning tree, the logs of this type can be parsed into structured information with readability.

[0080] Step 105, parsing the logs of the target clustering type according to the parsing template spanning tree to obtain a log parsing result.

[0081] In this step, after determining the target clustering type of the log to be parsed, the parsing template spanning tree corresponding to the target clustering type can be used for parsing to obtain the log parsing result of the log to be parsed, which is the structured information with readability.

[0082] As an optional embodiment, the constructing the hierarchical tree model corresponding to the feature information based on the minimum spanning tree comprises:

[0083] The minimum step parameter and the field radius parameter in the density-based clustering method DBSCAN with noise are optimized by constructing the hierarchical tree model corresponding to the feature information based on the minimum spanning tree.

[0084] The clustering processing on the logs in the log data set to be parsed based on the hierarchical tree model, and obtaining the respective tree structure corresponding to each clustering type in the clustering result, comprises:

[0085] perform clustering processing on logs in the to-be-resolved log data set based on the optimized DBSCAN to obtain a clustering result;

[0086] obtain a respective tree structure corresponding to each type of log in the clustering result.

[0087] In some embodiments, the feature information corresponding to the hierarchical tree model is constructed by using a minimum spanning tree, the minimum step parameter and the field radius parameter in the density-based clustering method DBSCAN with noise are optimized, and the clustering processing is performed on the logs in the to-be-resolved log data set based on the optimized DBSCAN to obtain a clustering result, which can include the following processes:

[0088] 1) performing spatial transformation on the feature information of the logs according to the density and sparsity in the hierarchical tree model;

[0089] 2) constructing a minimum spanning tree of a distance-weighted graph based on the spatial transformation structure.

[0090] 3) constructing a cluster hierarchy structure of the associated points in the minimum spanning tree.

[0091] 4) constructing a hierarchical structure of the minimum clusters.

[0092] 5) extracting stable clusters from the compressed tree as the clustering result.

[0093] In the embodiment, the feature information corresponding to the hierarchical tree model is constructed by using a minimum spanning tree, the minimum step parameter and the field radius parameter in the density-based clustering method DBSCAN with noise are optimized, so that the accuracy and the applicable range of the optimized DBSCAN are improved, and the manual adjustment of the minimum step parameter and the field radius parameter in the clustering process is avoided, thereby improving the automation degree of the clustering process.

[0094] In some embodiments, the clustering result is determined based on the mutual reachable distance between the logs.

[0095] For example, the distance between the feature information of different logs can be calculated according to the following formula in the clustering algorithm:

[0096] ;

[0097] wherein, d mr-k (A-B) is the mutual reachable distance between A and B; d AB is the Euclidean distance between A and B; d corek (A) is the distance from A to the core point of the clustering cluster; and d corek(B) is the distance from B to the core point of the cluster, wherein A and B represent the feature information of two logs respectively, such as the second feature vectors of the two logs respectively.d mr-k The maximum value of (A-B) and d corek (A), d corek (B), and d AB .

[0098] In this way, by improving the distance calculation in the clustering process, the distance from A and B to the core point of each cluster, and the Euclidean distance between A and B are fused in the clustering process, so that the clustering result is more accurate.

[0099] As an optional implementation, the generating tree processing of the logs of the target clustering type according to the target clustering type and the target tree structure corresponding to the target clustering type, to obtain the parsing template generation tree of the target clustering type, comprises:

[0100] performing second tokenization processing on the logs belonging to the target clustering type in the to-be-parsed log data set to obtain a token sequence of each log of the target clustering type, wherein the token sequence comprises L tokens, L is an integer greater than 1, and the tokens in the token sequence comprise first tokens common to the logs of the target clustering type and second tokens not common to the logs of the target clustering type;

[0101] inserting the tokens in the target token sequence into the target tree structure from the root node of the target tree structure corresponding to the target clustering type as a starting point to obtain the parsing template generation tree of the target clustering type, wherein the target token sequence comprises the token sequence of each log of the target clustering type.

[0102] It is worth mentioning that the second tokenization processing in the embodiment needs to be distinguished from the first tokenization processing in the foregoing embodiment, wherein the first tokenization processing is implemented based on a regular tokenizer or a dictionary, and the segmented tokens after tokenization processing do not contain character segments of pure symbols or pure numbers. However, the second tokenization processing does not have this requirement.

[0103] For example, taking the following email service log as an example:

[0104] Oct 6 00:00:13 localhost postfix / smtpd

[26072] :

[0105] disconnect from n169-113.mail.139.com[120.232.169.113] ehlo=1 mail=1rcpt=9 / 10 data=1 quit=1 commands=13 / 14.

[0106] In the above example, [120.232.169.113] is a character segment composed of pure numbers and symbols, which does not belong to the segmented words after the first segmentation processing, but belongs to the segmented words after the second segmentation processing.

[0107] In other words, the log content can be roughly divided into two parts, namely, a fixed part and a variable parameter part, wherein the fixed part is usually a segmented word with language meaning, and for the same type of log, the fixed part is the same, while the variable parameter part can be a character segment composed of pure numbers and symbols, and even for the same type of log, different variable parameters can be included.

[0108] For example, in the above email log, the segmented words belonging to the fixed part include 'disconnect from', 'ehlo=','mail=', 'rcpt=', 'data=', 'quit=', 'commands='; the segmented words belonging to the variable parameter part include 'n169-113.mail.139.com [120.232.169.113]', '1', '1', '9 / 10', '1', '1', '12 / 14'.

[0109] In some embodiments, starting from the root node of the target tree structure corresponding to the target cluster type, the segmented words in the target segmented word sequence are sequentially inserted into the target tree structure to obtain the parsing template generation tree of the target cluster type, and the purpose is to form the log template with the segmented words belonging to the fixed part in the log, i.e., "disconnect from *ehlo=* mail=* rcpt=* data=* quit=* commands=*", wherein the wildcard "*" represents the segmented words of the variable parameter part.

[0110] It should be noted that after obtaining the clustering result of the log, the second segmentation processing is performed again on all logs of the same cluster type, so that the segmented word result of each log has the same number of segmented words, i.e., the lengths of the segmented word sequences of different logs of the same cluster type are the same.

[0111] For example, the segmented word sequence of the above email log is as follows:

[0112] ['disconnect', 'from', 'n169-113.mail.139.com', '[120.232.169.113]', 'ehlo', '=', '1','mail', '=', '1', 'rcpt', '=', '9 / 10', 'data', '=', '1', 'quit', '=', '1', 'commands', '=', '13 / 14'].

[0113] It should be noted that, starting from the root node of the target tree structure corresponding to the target cluster type, the words in the target word sequence are inserted into the target tree structure in turn, which can be according to the arrangement order of the words in the target word sequence, starting from the first word in the target word sequence, each node in the target tree structure is matched, and the adjacent nodes in the target tree structure are also adjacent in the range of searching.

[0114] In some embodiments, the target tree structure corresponding to the target cluster type is obtained by starting from the root node of the target tree structure, and inserting the words in the target word sequence into the target tree structure in turn, including:

[0115] Starting from the root node of the target tree structure corresponding to the target cluster type, searching for nodes in the target tree structure that match the words in the target word sequence;

[0116] According to the search result, updating the target tree structure;

[0117] According to the updated target tree structure, determining the parsing template generation tree of the target cluster type.

[0118] In some embodiments, the search result can include the target tree structure that exists in the target word sequence and the target tree structure that does not exist in the target word sequence, and at this time, according to the search result, the target tree structure can be updated, when the search result is the target tree structure that exists in the target word sequence, the target tree structure is not updated, and the next word in the target word sequence is searched in the target tree structure, until all words in the target word sequence are searched; of course, when the search result is the target tree structure that does not exist in the target word sequence, the target tree structure needs to be updated, for example: a branch can be added in the target tree structure according to the word that does not find the matching node.

[0119] In some implementations, a node that matches a segment in the target segmentation sequence may be one whose stored characters are the same as the segment; a node that does not match a segment in the target segmentation sequence may be one whose stored characters are different from the segment.

[0120] In this implementation, the target tree structure can be updated by matching the word segmentation results in the target tree structure with the target word segmentation sequence, and finally a parsing template generation tree that can match the fixed part of the log of this type can be obtained.

[0121] As an optional implementation, updating the target tree structure based on the search results includes:

[0122] If the target tree structure is determined to include a first node that matches the third word, then in the subtree corresponding to the first node, a node that matches the fourth word in the target word sequence is searched, wherein the target word sequence includes the third word and the fourth word, and the fourth word is the next word in the target word sequence that is adjacent to the third word.

[0123] If it is determined that the target tree structure does not include a node that matches the fifth word, a branch corresponding to the fifth word is added to the subtree corresponding to the second node in the target tree structure. The target word segmentation sequence includes the fifth word and the sixth word, and the sixth word is the word preceding the fifth word in the target word segmentation sequence. The second node matches the sixth word.

[0124] With a log entry m c For example, the word segmentation sequence obtained after the second word segmentation process is T. m The step of inserting the words in the target word segmentation sequence into the target tree structure sequentially, starting from the root node of the target tree structure corresponding to the target clustering type, may include the following process:

[0125] For the word segmentation sequence T m Perform a traversal and extract the word t. m Insert the data sequentially into the target tree structure. The special character '$' in the target tree structure is the marker for the last node of each branch. Therefore, the path from any '$' to the root node represents a log entry m. c The matching path, where the depth of the target tree structure is L+1, and the word segmentation sequence T m The number of word segments in the text is L;

[0126] Word segmentation sequence T m The word t in m First, traverse every path in the target tree structure. Start searching from the root node P to obtain the path that is to be searched and the word segmentation t.mi The matched node, if there is no matched node, the word sequence T m Inserts the tree structure as a new branch; if there is a matched node, selects the subtree in the target tree structure corresponding to the matched node, and goes to the subtree to continue the word sequence T m The next word in the word sequence T m The word t m All the searches are completed.

[0127] For example, based on the word sequence of the above email log: ['disconnect', 'from', 'n169-113.mail.139.com', '[120.232.169.113]', 'ehlo', '=', '1','mail', '=', '1', 'rcpt', '=', '9 / 10', 'data', '=', '1', 'quit', '=', '1', 'commands', '=', '13 / 14', ], the process of constructing the target tree structure is as shown in Figure 2 , wherein the initial target tree structure is as shown in the left tree structure in Figure 2 The initial target tree structure can be determined based on the number of words contained in the word sequence of a log of the target cluster type, and thereafter, in the process of inserting the first word sequence T m , if the second word t m2 in the sequence is found to be the same as the word stored in the 3rd node Q in the target tree structure, P is updated to Q, and the next word t m in the word sequence T m3 is searched starting from Q. If t m3 fails to match the 4th node in the target tree structure, the node of the matching failure is recorded, the word t m3 is inserted into the 4th node in the target tree structure as a new branch, the root node P is updated to the 4th node, and the third word t m in the word sequence T m3 is continued to match the 5th node in the target tree structure until the search of all the words in the entire word sequence T m is completed, and the search process of a log is completed. Then, the next log is inserted in the target tree structure, and the insertion of all the logs of the target cluster type is completed to obtain the final parsing template generation tree.

[0128] In the embodiment, the segmented sequences of the same cluster type log can be sequentially inserted into the target tree structure to obtain a parsing template generation tree that can reflect the parsing rule of the log of the cluster type. In this way, the parsing template generation tree can implement the parsing process of the log of the cluster type.

[0129] For example, in the process of parsing the log using the parsing template generation tree, the characters in the log can be sequentially matched with the nodes in the parsing template generation tree, and the t m , thereby obtaining a structured log template and completing the log parsing process.

[0130] As shown in FIG. 8, for the segmentation of the fixed part, there is usually only one branch, and for the segmentation of the variable parameter part, there are multiple branches. Figure 2

[0131] In some embodiments, the target tree structure can also be verified, and in the case where the target tree structure passes the verification, the adding of branches in the target tree structure can be stopped. In this way, in the case where the parsing degree of the target tree structure on the log meets the requirements, the branches in the target tree structure can not be added based on the log, the complexity of the structure of the target tree structure can be simplified in the case where it is ensured that the target tree structure can meet the parsing degree requirements of the same type of log, the complexity of the finally obtained parsing template generation tree can be further simplified, and the parsing efficiency and training efficiency of the parsing template generation tree can be improved.

[0132] As an optional embodiment, the method further includes:

[0133] obtaining a first parameter and a second parameter, wherein the first parameter is the number of segmented words that are not found to have a matching node in the target tree structure, and the second parameter is the number of wildcard symbols contained in the path for finding each segmented word in the target segmented sequence in the target tree structure;

[0134] determining an index parameter of the target tree structure according to the first parameter, the second parameter, and the length of the target segmented sequence;

[0135] In the case where it is determined that the target tree structure does not include a node matching the fifth segmented word, a branch corresponding to the fifth segmented word is added in a subtree corresponding to the second node in the target tree structure, and the method includes the following steps.

[0136] In the case where the index parameter meets a first preset condition, a branch corresponding to the third segmented word is added in a subtree corresponding to the fourth node in the target tree structure; or the method further includes: in the case where the index parameter does not meet the first preset condition, a node corresponding to the third segmented word in the target tree structure is updated to a wildcard symbol. ​

[0137] In some embodiments, in the case of sequentially searching for the word pieces in the word piece sequence in the target tree structure, the node that matches successfully feeds back the word piece stored in the node, and the node that fails to match returns a wildcard symbol '*'. In this way, an index parameter for judging whether the target tree structure can meet the analysis degree requirement of the same type of log can be calculated according to the number of word pieces contained in a complete log, the number of wildcards '*' returned by the target tree structure, and the number of wildcard symbols contained in the matching path of the target tree structure for searching the word piece sequence.

[0138] If the index parameter meets a first preset condition, it indicates that the target tree structure cannot meet the analysis degree requirement of the same type of log; if the index parameter does not meet the first preset condition, it indicates that the target tree structure can meet the analysis degree requirement of the same type of log.

[0139] For example, the index parameter can be calculated by the following formula:

[0140] X =∑unmatch(t mi ) / L-θ;

[0141] Wherein, L represents the length of the target word piece sequence; ∑unmatch(t mi ) represents the first parameter; θ represents the second parameter; and the first condition is X≥λ.

[0142] In this way, when X≥λ, a branch corresponding to the third word piece is added in the subtree corresponding to the fourth node in the target tree structure; when X<λ, a branch corresponding to the third word piece is not added in the subtree corresponding to the fourth node in the target tree structure.

[0143] In the case where the analysis degree of the target tree structure on the log meets the requirement, the present embodiment can increase the branch in the target tree structure based on the log, can simplify the structural complexity of the target tree structure in the case where the target tree structure can meet the analysis degree requirement of the same type of log, and further simplify the complexity of the analysis template generation tree finally obtained, and improve the analysis efficiency and training efficiency of the analysis template generation tree.

[0144] In the embodiments of the present application, feature information of each log in a to-be-analyzed log data set is acquired, a hierarchical tree model corresponding to the feature information is constructed based on a minimum spanning tree, and the logs in the to-be-analyzed log data set are processed based on the hierarchical tree model, and a tree structure corresponding to each of the clustering types in the clustering result is acquired according to the clustering result. According to a target clustering type and a target tree structure corresponding to the target clustering type, the logs of the target clustering type are processed based on a spanning tree, and a parsing template spanning tree of the target clustering type is obtained, wherein the clustering types in the clustering result include the target clustering type. The logs of the target clustering type are parsed based on the parsing template spanning tree, and a log parsing result is obtained. After the feature information of the logs is acquired, the hierarchical tree model corresponding to the feature information is constructed based on the minimum spanning tree, and the logs in the to-be-analyzed log data set are processed based on the hierarchical tree model, and finally, the parsing template spanning tree is automatically generated for the logs of each clustering type as the parsing rule applicable to the logs of the clustering type based on the spanning tree technology, so as to realize efficient parsing of various logs. The method can realize automation of the whole log parsing process, and significantly improve the log parsing efficiency. Based on the idea of hierarchical clustering, the clustering process is optimized, so that the hierarchical tree model can automatically obtain the optimal clustering result only by giving the minimum sample contained in the cluster, and the process of manual parameter adjustment is avoided. On the basis of improving the automation of the log parsing process, the accuracy and application range of the hierarchical tree model are also improved.

[0145] Referring to Figure 3 , the embodiments of the present application also provide a log parsing method, as shown in Figure 3 , the log parsing method comprises the following steps:

[0146] Step 301, a large amount of log data is acquired, and the log data is preprocessed.

[0147] In some embodiments, the preprocessing in this step can include at least one of the following:

[0148] The log data is cleaned to remove error records, duplicate records, blank lines, comments, and duplicate log entries;

[0149] The log data of different sources is uniformly formatted;

[0150] The abnormal values are modified or deleted;

[0151] The null values are filled or deleted.

[0152] In some embodiments, for the abnormal values in the log data, a label can be marked, and the logs with abnormal values are separately taken as a kind of log type, clustering, constructing the parsing template generation tree, so as to realize the parsing of the abnormal values by using the parsing template generation tree. In this way, compared with the foregoing manner of modifying or deleting the abnormal values, the parsing of the abnormal values can be realized, and the omission of the parsing of the abnormal values can be avoided.

[0153] In some embodiments, the log data set to be parsed can be acquired regularly, such as once every 1 day or 1 week.

[0154] In some embodiments, the computing power load of the electronic device performing the log parsing method of the embodiments of the present application can be detected, and the log parsing process can be performed in a time period when the computing power load of the electronic device is low, for example, log parsing is performed at 2:00 in the morning every day.

[0155] Step 302, extracting log features by using a natural language processing (NLP) technology, and converting the log into a feature vector.

[0156] In some embodiments, the log can be subjected to first segmentation processing based on a regular segmenter or a dictionary, U segmented words of the log are obtained, a first feature vector is determined according to feature values of the U segmented words, and finally, the first feature vector is subjected to dimension reduction processing to obtain a second feature vector.

[0157] The dimension reduction processing of the first feature vector can be to reduce the dimension of the first feature vector by using a t-SNE algorithm, so as to avoid the overfitting problem caused by using the high-dimensional first feature vector for clustering processing.

[0158] Step 303, clustering the obtained feature vector by using a DBSCAN algorithm.

[0159] In this step, the DBSCAN clustering algorithm can be improved based on the idea of a hierarchical tree model, so that the improved DBSCAN algorithm can automatically obtain the optimal clustering result only by being given the minimum samples contained in the cluster, and the complicated parameter adjustment process is avoided.

[0160] Step 304, generating a corresponding parsing template generation tree for the logs of different clustering types.

[0161] In this step, the tree structure corresponding to each clustering type of the logs can be generated by using the tree generation method according to the clustering result, and the tree structure is updated based on the logs of the same type to obtain the parsing template generation tree of each clustering type.

[0162] Step 305, generating structured data.

[0163] In this step, the log of each cluster type is parsed by using the parsing template of the corresponding generation tree of each cluster type to obtain the structured data of each log in the to-be-parsed log data set, and the parsing of each log in the to-be-parsed log data set is completed.

[0164] The log parsing method has at least the following beneficial effects:

[0165] Compared with the way of manually configuring a log parsing file in the related art and compared with the log parsing method that is partially automated in the related art, the present application realizes the automation of the whole log parsing process, can improve the parsing efficiency of log parsing, and saves labor costs;

[0166] Compared with the log parsing process in the related art, the present application improves the calculation method of the feature vector and the clustering method, can improve the efficiency and applicability of the clustering process, and reduces the complexity of parameter adjustment of the clustering algorithm;

[0167] Compared with the way of pre-setting a parsing rule for log parsing in the related art, the present application classifies logs in advance before parsing, improves the efficiency in the parsing process, and reduces the consumption of computing resources.

[0168] The present application also provides a log parsing device. Referring to Figure 4 , Figure 4 is a structural diagram of the log parsing device 400 provided by the present application. Since the log parsing device 400 solves the problem by the similar principle as the log parsing method in the present application, the implementation of the log parsing device can be referred to the implementation of the method, and the repeated parts will not be described herein.

[0169] As shown in Figure 4 , the log parsing device 400 comprises:

[0170] The first acquisition module 401 is configured to acquire the feature information of each log in the to-be-parsed log data set;

[0171] The model construction module 402 is configured to construct a hierarchical tree model corresponding to the feature information based on a minimum spanning tree;

[0172] The clustering module 403 is configured to perform clustering processing on the logs in the to-be-parsed log data set based on the hierarchical tree model, and acquire the tree structure corresponding to each cluster type in the clustering result according to the clustering result;

[0173] The spanning tree module 404 is configured to perform spanning tree processing on the logs of the target cluster type according to a target cluster type and a target tree structure corresponding to the target cluster type, to obtain an analysis template spanning tree of the target cluster type, wherein the cluster types in the cluster result include the target cluster type.

[0174] The log analysis module 405 is configured to analyze the logs of the target cluster type according to the analysis template spanning tree, to obtain a log analysis result.

[0175] Optionally, the first acquisition module 401 is specifically configured to:

[0176] perform feature extraction processing on each log in the log data set to be analyzed, to obtain feature information of the each log;

[0177] The feature extraction processing includes:

[0178] perform first segmentation processing on the log based on a regular segmenter or a dictionary, to obtain U segmented words of the log, U being an integer greater than 1;

[0179] determine a first feature vector according to feature values of the U segmented words, the first feature vector being a U-dimensional vector;

[0180] perform dimension reduction processing on the first feature vector, to obtain a second feature vector, the feature information of the each log including the second feature vector, the second feature vector being a two-dimensional feature vector.

[0181] Optionally, the model construction module 402 is specifically configured to:

[0182] construct a hierarchical tree model corresponding to the feature information by using a minimum spanning tree, and optimize a minimum step parameter and a field radius parameter in a density-based clustering method DBSCAN with noise;

[0183] The clustering module 403 includes:

[0184] The clustering processing unit is configured to perform clustering processing on the logs in the log data set to be analyzed based on the optimized DBSCAN, to obtain a cluster result;

[0185] The acquisition unit is configured to acquire a tree structure corresponding to each cluster type in the cluster result.

[0186] Optionally, the cluster result is determined based on mutual reachable distances between the logs.

[0187] Optionally, the spanning tree module 404 includes:

[0188] The word segmentation unit is configured to perform second word segmentation processing on the logs belonging to the target clustering type in the log data set to be parsed, to obtain a word segmentation sequence of each log of the target clustering type, wherein the word segmentation sequence includes L words, L is an integer greater than 1, and the words in the word segmentation sequence include first words common to the logs of the target clustering type and second words not common to the logs of the target clustering type.

[0189] The insertion unit is configured to insert the words in a target word segmentation sequence into a target tree structure corresponding to the target clustering type, starting from a root node of the target tree structure, to obtain a parsing template generation tree of the target clustering type, wherein the target word segmentation sequence includes the word segmentation sequence of each log of the target clustering type.

[0190] Optionally, the insertion unit includes:

[0191] The searching sub-unit is configured to search, starting from a root node of a target tree structure corresponding to the target clustering type, a node in the target tree structure that matches a word in a target word segmentation sequence.

[0192] The updating sub-unit is configured to update the target tree structure according to the searching result.

[0193] The determining sub-unit is configured to determine the parsing template generation tree of the target clustering type according to the updated target tree structure.

[0194] Optionally, the updating sub-unit is specifically configured to:

[0195] In a case where it is determined that the target tree structure includes a first node that matches a third word, the searching sub-unit is configured to search, in a sub-tree corresponding to the first node, a node that matches a fourth word in the target word segmentation sequence, wherein the target word segmentation sequence includes the third word and the fourth word, and the fourth word is a next word adjacent to the third word in the target word segmentation sequence.

[0196] In a case where it is determined that the target tree structure does not include a node that matches a fifth word, the adding sub-unit is configured to add, in a sub-tree corresponding to a second node in the target tree structure, a branch corresponding to the fifth word, wherein the target word segmentation sequence includes the fifth word and a sixth word, and the sixth word is a previous word adjacent to the fifth word in the target word segmentation sequence, and the second node matches the sixth word.

[0197] Optionally, the log parsing apparatus 400 further includes:

[0198] The second obtaining module is configured to obtain a first parameter and a second parameter, where the first parameter is a number of segmented words that are not found in the target tree structure, and the second parameter is a number of wildcard symbols contained in a path for finding each segmented word in the target segmented word sequence in the target tree structure.

[0199] The determining module is configured to determine an index parameter of the target tree structure according to the first parameter, the second parameter, and a length of the target segmented word sequence.

[0200] The updating subunit is further configured to:

[0201] In a case where the index parameter satisfies a first preset condition, add a branch corresponding to the third segmented word in a sub-tree corresponding to a fourth node in the target tree structure; or the log analysis device 400 further includes an updating module configured to, in a case where the index parameter does not satisfy the first preset condition, update a node corresponding to the third segmented word in the target tree structure into a wildcard symbol.

[0202] The log analysis device 400 provided by the embodiments of the present application can execute the method embodiments described above, and the implementation principles and technical effects are similar, and the embodiments are not described here again.

[0203] The embodiments of the present application further provide an electronic device. Since the principle of solving problems by the electronic device is similar to the log analysis method in the embodiments of the present application, the implementation of the electronic device can be referred to the implementation of the method, and the repeated parts are not described here again. As shown in Figure 5 The electronic device of the embodiments of the present application includes a processor 500 configured to read a program in a memory 520 and execute the following processes:

[0204] Obtain feature information of each log in a log data set to be analyzed;

[0205] Construct a hierarchical tree model corresponding to the feature information based on a minimum spanning tree;

[0206] Perform clustering processing on logs in the log data set to be analyzed based on the hierarchical tree model, and obtain a tree structure corresponding to each clustering type in a clustering result according to the clustering result;

[0207] Perform a spanning tree processing on logs of a target clustering type according to the target clustering type and a target tree structure corresponding to the target clustering type, to obtain an analysis template spanning tree of the target clustering type, where the clustering types in the clustering result include the target clustering type;

[0208] Perform analysis on logs of the target clustering type according to the analysis template spanning tree, to obtain a log analysis result.

[0209] wherein, in Figure 5 The bus architecture can include any number of interconnected buses and bridges, specifically, various circuitry linking the one or more processors represented by the processor 500 and the memory represented by the memory 520. The bus architecture can also link various other circuitry such as peripheral devices, voltage regulators, and power management circuitry, which are well known in the art and thus, are not further described herein. The bus interface provides an interface. The processor 500 is responsible for managing the bus architecture and general processing, and the memory 520 can store data used by the processor 500 in performing operations.

[0210] Optionally, the processor 500 is further configured to read a program in the memory 520 and perform the following steps:

[0211] performing feature extraction processing on each log in the to-be-analyzed log data set respectively to obtain feature information of the each log;

[0212] The feature extraction processing includes:

[0213] performing first segmentation processing on the log based on a regular segmenter or a dictionary to obtain U segmented words of the log, U being an integer greater than 1;

[0214] determining a first feature vector according to feature values of the U segmented words, the first feature vector being a U-dimensional vector;

[0215] performing dimension reduction processing on the first feature vector to obtain a second feature vector, the feature information of the each log including the second feature vector, the second feature vector being a two-dimensional feature vector.

[0216] Optionally, the processor 500 is further configured to read a program in the memory 520 and perform the following steps:

[0217] adopting a minimum spanning tree to construct a hierarchical tree model corresponding to the feature information, and optimizing a minimum step parameter and a field radius parameter in a density-based clustering method DBSCAN with noise;

[0218] performing clustering processing on logs in the to-be-analyzed log data set based on the hierarchical tree model, and obtaining, according to a clustering result, a tree structure corresponding to each clustering type in the clustering result, including:

[0219] performing clustering processing on logs in the to-be-analyzed log data set based on the optimized DBSCAN to obtain a clustering result;

[0220] obtaining a tree structure corresponding to logs of each clustering type in the clustering result.

[0221] Optionally, the clustering result is determined based on mutual reachable distances between logs.

[0222] Optionally, the processor 500 is further configured to read a program in the memory 520 and perform the following steps:

[0223] performing a second word segmentation processing on the logs in the to-be-analyzed log data set that belong to the target clustering type, to obtain a word segmentation sequence of each log of the target clustering type, wherein the word segmentation sequence includes L words, L is an integer greater than 1, and the words in the word segmentation sequence include first words common to the logs of the target clustering type and second words not common to the logs of the target clustering type;

[0224] inserting the words in the target word segmentation sequence into the target tree structure successively from a root node of a target tree structure corresponding to the target clustering type, to obtain an analysis template generation tree of the target clustering type, wherein the target word segmentation sequence includes the word segmentation sequence of each log of the target clustering type.

[0225] Optionally, the processor 500 is further configured to read a program in the memory 520 and perform the following steps:

[0226] finding, in the target tree structure, a node matching a word in the target word segmentation sequence from a root node of a target tree structure corresponding to the target clustering type;

[0227] updating the target tree structure according to the finding result;

[0228] determining an analysis template generation tree of the target clustering type according to the updated target tree structure.

[0229] Optionally, the processor 500 is further configured to read a program in the memory 520 and perform the following steps:

[0230] finding, in a subtree corresponding to the first node, a node matching a fourth word in the target word segmentation sequence in a case where it is determined that the target tree structure includes the first node matching the third word, wherein the target word segmentation sequence includes the third word and the fourth word, and the fourth word is a next word adjacent to the third word in the target word segmentation sequence;

[0231] adding, in a subtree corresponding to a second node in the target tree structure, a branch corresponding to a fifth word in a case where it is determined that the target tree structure does not include a node matching the fifth word, wherein the target word segmentation sequence includes the fifth word and a sixth word, and the sixth word is a previous word adjacent to the fifth word in the target word segmentation sequence, and the second node matches the sixth word.

[0232] Optionally, the processor 500 is further configured to read a program in the memory 520 and perform the following steps:

[0233] obtain a first parameter and a second parameter, wherein the first parameter is a number of segmented words that are not found in the target tree structure, and the second parameter is a number of wildcard symbols contained in a path for finding each segmented word in the target segmented word sequence in the target tree structure;

[0234] determine an index parameter of the target tree structure according to the first parameter, the second parameter, and a length of the target segmented word sequence;

[0235] in a case where the index parameter meets a first preset condition, add a branch corresponding to the third segmented word in a subtree corresponding to a fourth node in the target tree structure; or in a case where the index parameter does not meet the first preset condition, update a node corresponding to the third segmented word in the target tree structure into a wildcard symbol.

[0236] The terminal provided in the embodiment of the application can execute the method embodiments, and has similar implementation principles and technical effects, which will not be described here again.

[0237] In addition, the computer readable storage medium of the embodiment of the application is used for storing a computer program, and the computer program can be executed by a processor to implement the following steps:

[0238] obtain feature information of each log in a log data set to be parsed;

[0239] construct a hierarchical tree model corresponding to the feature information based on a minimum spanning tree;

[0240] perform clustering processing on logs in the log data set to be parsed based on the hierarchical tree model, and obtain a tree structure corresponding to each clustering type in a clustering result according to the clustering result;

[0241] perform a spanning tree processing on logs of a target clustering type based on the target clustering type and a target tree structure corresponding to the target clustering type, to obtain a parsing template spanning tree of the target clustering type, wherein the clustering types in the clustering result include the target clustering type;

[0242] perform parsing on logs of the target clustering type based on the parsing template spanning tree, to obtain a log parsing result.

[0243] Optionally, the computer program can also be executed by the processor to implement the following steps:

[0244] respectively performing feature extraction processing on each log in the to-be-analyzed log data set to obtain feature information of the each log;

[0245] The feature extraction processing comprises:

[0246] performing first segmentation processing on the log based on a regular segmenter or a dictionary to obtain U segmented words of the log, U being an integer greater than 1;

[0247] determining a first feature vector according to feature values of the U segmented words, the first feature vector being a U-dimensional vector;

[0248] performing dimension reduction processing on the first feature vector to obtain a second feature vector, the feature information of the each log comprising the second feature vector, the second feature vector being a two-dimensional feature vector.

[0249] Optionally, the computer program can also be executed by the processor to implement the following steps:

[0250] adopting a minimum spanning tree to construct a hierarchical tree model corresponding to the feature information, and optimizing a minimum step parameter and a field radius parameter in a density-based clustering method DBSCAN with noise;

[0251] performing clustering processing on the logs in the to-be-analyzed log data set based on the hierarchical tree model, and obtaining, according to a clustering result, a tree structure corresponding to each clustering type in the clustering result, comprising:

[0252] performing clustering processing on the logs in the to-be-analyzed log data set based on the optimized DBSCAN to obtain a clustering result;

[0253] obtaining a tree structure corresponding to each clustering type of logs in the clustering result.

[0254] Optionally, the clustering result is determined based on mutual reachable distances between logs.

[0255] Optionally, the computer program can also be executed by the processor to implement the following steps:

[0256] performing second segmentation processing on logs belonging to a target clustering type in the to-be-analyzed log data set to obtain a segmentation sequence of each log of the target clustering type, wherein the segmentation sequence comprises L segmented words, L being an integer greater than 1, and the segmented words in the segmentation sequence comprise first segmented words common to logs of the target clustering type and second segmented words not common to logs of the target clustering type;

[0257] Start from a root node of a target tree structure corresponding to the target cluster type, and sequentially insert the wordpieces in the target wordpiece sequence into the target tree structure to obtain a parsing template generation tree of the target cluster type, wherein the target wordpiece sequence includes a wordpiece sequence of each log of the target cluster type.

[0258] Optionally, the computer program can also be executed by the processor to implement the following steps:

[0259] Start from a root node of a target tree structure corresponding to the target cluster type, and sequentially insert the wordpieces in the target wordpiece sequence into the target tree structure to obtain a parsing template generation tree of the target cluster type, wherein the target wordpiece sequence includes a wordpiece sequence of each log of the target cluster type.

[0260] According to the search result, update the target tree structure;

[0261] According to the updated target tree structure, determine the parsing template generation tree of the target cluster type.

[0262] Optionally, the computer program can also be executed by the processor to implement the following steps:

[0263] In a case where it is determined that the target tree structure includes a first node matched with a third wordpiece, find a node matched with a fourth wordpiece in a sub-tree corresponding to the first node, wherein the target wordpiece sequence includes the third wordpiece and the fourth wordpiece, and the fourth wordpiece is a next wordpiece adjacent to the third wordpiece in the target wordpiece sequence.

[0264] In a case where it is determined that the target tree structure does not include a node matched with a fifth wordpiece, add a branch corresponding to the fifth wordpiece in a sub-tree corresponding to a second node in the target tree structure, wherein the target wordpiece sequence includes the fifth wordpiece and a sixth wordpiece, and the sixth wordpiece is a previous wordpiece adjacent to the fifth wordpiece in the target wordpiece sequence, and the second node is matched with the sixth wordpiece.

[0265] Optionally, the computer program can also be executed by the processor to implement the following steps:

[0266] Obtain a first parameter and a second parameter, wherein the first parameter is a number of wordpieces that are not found to be matched in the target tree structure, and the second parameter is a number of wildcard symbols contained in a path for finding each wordpiece in the target wordpiece sequence in the target tree structure;

[0267] According to the first parameter, the second parameter, and a length of the target wordpiece sequence, determine an index parameter of the target tree structure.

[0268] In a case where the index parameter meets a first preset condition, a branch corresponding to the third word segmentation is added in a subtree corresponding to a fourth node in the target tree structure; or in a case where the index parameter does not meet the first preset condition, a node corresponding to the third word segmentation in the target tree structure is updated as a wildcard.

[0269] The embodiments of the present application also provide a computer program product comprising computer instructions, which, when executed by a processor, implement each step of the aforementioned log analysis method embodiments and can achieve the same beneficial effects as the aforementioned log analysis method embodiments. To avoid repetition, no further elaboration is provided here.

[0270] In several embodiments provided in the present application, it should be understood that the disclosed method and device can be implemented in other manners. For example, the described device embodiments are merely schematic, and the division of the units is merely a logical function division. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0271] In addition, each function unit in the embodiments of the present application can be integrated into a processing unit, or each unit can be a separate physical unit, or two or more units can be integrated into a unit. The above integrated unit can be implemented in the form of hardware, or can be implemented in the form of hardware plus software function units.

[0272] The integrated unit implemented in the form of software function units can be stored in a computer readable storage medium. The above software function unit stored in a storage medium includes a plurality of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute part of the steps of the transceiving method described in the embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0273] The above is the preferred embodiment of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered within the scope of protection of the present application.

Claims

1. A log parsing method, characterized in that, include: Obtain the feature information of each log entry in the log dataset to be parsed; Construct a hierarchical tree model corresponding to the feature information based on the minimum spanning tree; Based on the hierarchical tree model, the logs in the log dataset to be parsed are clustered, and the tree structure corresponding to each clustering type in the clustering results is obtained according to the clustering results. Based on the target clustering type and the target tree structure corresponding to the target clustering type, the logs of the target clustering type are subjected to spanning tree processing to obtain the parsing template spanning tree of the target clustering type, wherein the clustering type in the clustering result includes the target clustering type; The logs of the target cluster type are parsed using the parsing template to generate a tree, and the log parsing results are obtained. The process of obtaining the feature information of each log entry in the log dataset to be parsed includes: Each log entry in the log dataset to be parsed is subjected to feature extraction processing to obtain the feature information of each log entry; The feature extraction process includes: Based on a regular expression segmenter or dictionary, the log is processed by the first segmentation to obtain U segmented words of the log, where U is an integer greater than 1; Based on the feature values ​​of the U segmented words, a first feature vector is determined, wherein the first feature vector is a U-dimensional vector; The first feature vector is reduced in dimensionality to obtain a second feature vector. The feature information of each log entry includes the second feature vector, which is a two-dimensional feature vector. The construction of the hierarchical tree model corresponding to the feature information based on the minimum spanning tree includes: A hierarchical tree model corresponding to the feature information is constructed using a minimum spanning tree, and the minimum step size parameter and neighborhood radius parameter in the density-based clustering method DBSCAN with noise are optimized. The process of clustering logs in the log dataset to be parsed based on the hierarchical tree model, and obtaining the tree structure corresponding to each clustering type in the clustering results, includes: Based on the optimized DBSCAN, the logs in the log dataset to be parsed are clustered to obtain the clustering results; Obtain the tree structure corresponding to the logs for each clustering type in the clustering results.

2. The method according to claim 1, characterized in that, in, The clustering results are determined based on the mutual reachability distance between logs.

3. The method according to any one of claims 1 to 2, characterized in that, The step of performing a spanning tree process on the logs of the target cluster type based on the target cluster type and the target tree structure corresponding to the target cluster type to obtain the parsing template spanning tree of the target cluster type includes: The logs belonging to the target cluster type in the log dataset to be parsed are subjected to a second word segmentation process to obtain a word segmentation sequence for each log of the target cluster type. The word segmentation sequence includes L words, where L is an integer greater than 1. The words in the word segmentation sequence include the first word common to the logs of the target cluster type and the second word not common to the logs of the target cluster type. Starting from the root node of the target tree structure corresponding to the target cluster type, the words in the target word segmentation sequence are inserted into the target tree structure in sequence to obtain the parsing template generation tree of the target cluster type. The target word segmentation sequence includes the word segmentation sequence of each log of the target cluster type.

4. The method according to claim 3, characterized in that, Starting from the root node of the target tree structure corresponding to the target clustering type, the words in the target word segmentation sequence are sequentially inserted into the target tree structure to obtain the parsing template generation tree of the target clustering type, including: Starting from the root node of the target tree structure corresponding to the target clustering type, search for nodes in the target tree structure that match the words in the target word segmentation sequence; Update the target tree structure based on the search results; Based on the updated target tree structure, determine the parsing template generation tree for the target clustering type.

5. The method according to claim 4, characterized in that, The step of updating the target tree structure based on the search results includes: If the target tree structure is determined to include a first node that matches the third word, then in the subtree corresponding to the first node, a node that matches the fourth word in the target word sequence is searched, wherein the target word sequence includes the third word and the fourth word, and the fourth word is the next word in the target word sequence that is adjacent to the third word. If it is determined that the target tree structure does not include a node that matches the fifth word, a branch corresponding to the fifth word is added to the subtree corresponding to the second node in the target tree structure. The target word segmentation sequence includes the fifth word and the sixth word, and the sixth word is the word preceding the fifth word in the target word segmentation sequence. The second node matches the sixth word.

6. The method according to claim 5, characterized in that, The method further includes: Obtain a first parameter and a second parameter, wherein the first parameter is the number of word segments for which no matching node is found in the target tree structure, and the second parameter is the number of wildcard symbols contained in the path of each word segment in the target word segmentation sequence in the target tree structure; The index parameters of the target tree structure are determined based on the first parameter, the second parameter, and the length of the target word segmentation sequence; When it is determined that the target tree structure does not include a node matching the fifth word, adding a branch corresponding to the fifth word to the subtree corresponding to the second node in the target tree structure includes: If the indicator parameters meet the first preset condition, a branch corresponding to the third word is added to the subtree corresponding to the fourth node in the target tree structure; or, if the indicator parameters do not meet the first preset condition, the node corresponding to the third word in the target tree structure is updated to a wildcard symbol.

7. A log parsing device, characterized in that, include: The first acquisition module is used to acquire the feature information of each log in the log dataset to be parsed. The model building module is used to construct a hierarchical tree model corresponding to the feature information based on the minimum spanning tree; The clustering module is used to cluster the logs in the log dataset to be parsed based on the hierarchical tree model, and to obtain the tree structure corresponding to each clustering type in the clustering results. The spanning tree module is used to perform spanning tree processing on the logs of the target clustering type according to the target clustering type and the target tree structure corresponding to the target clustering type, so as to obtain the parsing template spanning tree of the target clustering type, wherein the clustering type in the clustering result includes the target clustering type; The log parsing module is used to parse logs of the target cluster type according to the parsing template to generate a tree, and obtain log parsing results; The process of obtaining the feature information of each log entry in the log dataset to be parsed includes: Each log entry in the log dataset to be parsed is subjected to feature extraction processing to obtain the feature information of each log entry; The feature extraction process includes: Based on a regular expression segmenter or dictionary, the log is processed by the first segmentation to obtain U segmented words of the log, where U is an integer greater than 1; Based on the feature values ​​of the U segmented words, a first feature vector is determined, wherein the first feature vector is a U-dimensional vector; The first feature vector is reduced in dimensionality to obtain a second feature vector. The feature information of each log entry includes the second feature vector, which is a two-dimensional feature vector. The construction of the hierarchical tree model corresponding to the feature information based on the minimum spanning tree includes: A hierarchical tree model corresponding to the feature information is constructed using a minimum spanning tree, and the minimum step size parameter and neighborhood radius parameter in the density-based clustering method DBSCAN with noise are optimized. The process of clustering logs in the log dataset to be parsed based on the hierarchical tree model, and obtaining the tree structure corresponding to each clustering type in the clustering results, includes: Based on the optimized DBSCAN, the logs in the log dataset to be parsed are clustered to obtain the clustering results; Obtain the tree structure corresponding to the logs for each clustering type in the clustering results.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the log parsing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps in the log parsing method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps in the log parsing method as described in any one of claims 1 to 6.

Images