Log pattern recognition method and system

Abstract

The present invention provides a log pattern recognition method and system, wherein the recognition method comprises: performing word segmentation on each collected log data, marking the word attributes of each word segmentation after word segmentation, and obtaining a word segmentation part-of-speech list for each log data; removing the word segmentations that match the pre-stored words and pre-stored word attributes in the common vocabulary database from the word segmentation part-of-speech list of each log data, and forming the word segmentation list of each log data with the remaining word segmentations; based on the word segmentation list of each log data, vectorizing each log data, and clustering each log data after vectorization; based on the result of clustering each log data, identifying the log pattern of each type of log data. The present invention can greatly improve the processing efficiency and the accuracy of the log data after quantization; without the need for manual labeling data, the log data can be better understood and the log pattern of each type of log data can be identified.

Description

Technical Field

[0001] This invention relates to the field of log data processing, and in particular to a log pattern recognition method and system. Background Art

[0002] In operations and maintenance (O&M) work, log data is a crucial source of information. However, the volume of log data in large systems is enormous; tens of thousands of log entries can be generated in a short period, making it very difficult for O&M personnel to utilize the information within the logs. In real-world production environments, log data often exhibits patterns. Categorizing logs into corresponding patterns reduces the difficulty for O&M personnel in utilizing the log data and helps them better understand the system's status. Current pattern recognition technologies are typically based on nonparametric methods, such as prefix trees, while some methods utilize classification techniques from machine learning for pattern recognition.

[0003] The volume of log data in current business systems is enormous. Non-parametric methods easily exceed their capacity to handle such large datasets, encountering limitations such as memory constraints and computational efficiency issues. Furthermore, they face limitations in recognizing different log types, such as purely English logs, purely Chinese logs, and mixed English and Chinese logs. For logs containing Chinese characters, semantic extraction becomes crucial, often proving ineffective with non-parametric methods. Machine information classification methods, on the other hand, require a large amount of labeled data. In practice, labeling is resource-intensive, and the labels for log patterns need constant updating as patterns change.

[0004] In view of the shortcomings of existing technologies, there is an urgent need for a log pattern recognition method that can solve different log data types, improve recognition efficiency, and does not rely on data tags. Summary of the Invention

[0005] This invention provides a log pattern recognition method and system to address the technical deficiencies in the prior art.

[0006] This invention provides a log pattern recognition method, comprising:

[0007] Each piece of log data collected is segmented into words, and the word attributes of each segmented word are labeled to obtain a word segmentation part-of-speech list for each piece of log data. The word segmentation part-of-speech list includes the word corresponding to each piece of log data and the word attributes corresponding to each word.

[0008] Remove the word segments from the word segmentation part-of-speech list of each log data that match the pre-stored words and their attributes in the common vocabulary database, and then form the word segmentation list of each log data with the remaining word segments.

[0009] Based on the word segmentation list of each log data, each log data is vectorized, and the vectorized log data are clustered.

[0010] Based on the clustering results of each log data, the log pattern of each type of log data is identified.

[0011] According to the log pattern recognition method provided by the present invention, the step of segmenting the collected log data into words includes:

[0012] Perform language-level judgment on each line of log data;

[0013] Log data identified as being in English will be segmented using spaces as delimiters;

[0014] Log data identified as being in Chinese and / or a mixture of Chinese and English will be segmented using the jieba word segmentation tool.

[0015] According to the log pattern recognition method provided by the present invention, the pre-stored words in the commonly used vocabulary database contain any or a combination of the following pre-stored word attributes:

[0016] Generic operational term attributes;

[0017] Non-operational word attributes of generic categories;

[0018] Commonly used terminology attributes related to the source device of the log data.

[0019] According to the log pattern recognition method provided by the present invention, the step of vectorizing each log data based on the word segmentation list of each log data includes:

[0020] The word segmentation list is used to filter out words that appear more than a preset threshold in the log data. Based on the filtered word segmentation list for each log entry, each log entry is vectorized using TF-IDF. Furthermore, the preset threshold can be set to 10 occurrences.

[0021] According to the log pattern recognition method provided by the present invention, the clustering of each vectorized log data includes:

[0022] Using a hierarchical clustering method, a hierarchical nested clustering tree is created by calculating the similarity between data points of different categories in each vectorized log data. In the nested clustering tree, data points of different categories are the lowest level of the nested clustering tree, and the top level of the nested clustering tree is the root node of a cluster.

[0023] According to the log pattern recognition method provided by the present invention, creating a hierarchical nested clustering tree includes:

[0024] S21. Group all the vectorized log data into a first primitive class cluster;

[0025] S22. According to the preset splitting rules, split the first original cluster to create a first nested clustering tree;

[0026] S23. Determine whether the first nested clustering tree meets the preset conditions. If yes, then treat the first nested clustering tree as a hierarchical nested clustering tree. If no, return to S22.

[0027] The step of splitting the first original cluster according to the preset splitting rules includes:

[0028] Within the same primary class cluster, obtain the distance between every two data points, and find the first and second data points with the greatest distance;

[0029] The first data point and the second data point are assigned to different first-class clusters and second-class clusters, respectively;

[0030] Obtain the remaining data points in the first original cluster and the first and second distances between the first and second data points. If the first distance is less than the second distance, assign the remaining data points to the first cluster; otherwise, assign the remaining data points to the second cluster.

[0031] According to the log pattern recognition method provided by the present invention, creating a hierarchical nested clustering tree includes:

[0032] S31. Treat each data point in all vectorized log data as a second primitive cluster;

[0033] S32. Merge the second original clusters according to the preset merging rules to create a second nested clustering tree;

[0034] S33. Determine whether the second nested clustering tree meets the preset conditions. If yes, then treat the second nested clustering tree as a hierarchical nested clustering tree. If no, return to S32.

[0035] The merging of the second original cluster according to the preset merging rules includes:

[0036] Calculate the distance between every two second primitive clusters, and find the two second primitive clusters with the smallest distance;

[0037] Merge the two second primitive clusters with the smallest distance into one cluster.

[0038] The present invention also provides a log pattern recognition system, comprising:

[0039] The word segmentation part-of-speech list determination module is used to segment the collected log data into words, annotate the word attributes of each segmented word, and obtain the word segmentation part-of-speech list for each log data. The word segmentation part-of-speech list includes the word corresponding to each log data and the word attributes corresponding to each word.

[0040] The word segmentation list determination module is used to remove word segments that match the pre-stored words and their attributes in the common vocabulary database from the word segmentation part-of-speech list of each log data, and to form the word segmentation list of each log data by the remaining word segments in each log data.

[0041] The clustering module is used to vectorize each log data based on the word segmentation list of each log data, and then cluster the vectorized log data.

[0042] The log pattern recognition module is used to identify the log pattern of each type of log data based on the results of clustering each log data.

[0043] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the log pattern recognition methods described above.

[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the log pattern recognition method described above.

[0045] The log pattern recognition method provided by this invention segments the collected log data into words, labels the word attributes of each segmented word to obtain a word segmentation part-of-speech list for each log data. Log pattern recognition removes common and generalized words from the log data through word attribute correspondence, which can significantly improve processing efficiency and the accuracy of quantified log data. Based on the word segmentation list of each log data, each log data is vectorized, and the vectorized log data are clustered, eliminating the need for manual data labeling. By using word segmentation tools and optimized TF-IDF, unstructured text data is transformed into vector data, quantifying the log text data and reflecting the semantic information of the logs, enabling a better understanding of the log data and identification of log patterns for each type of log data. Attached Figure Description

[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 This is a schematic flowchart of a log pattern recognition method provided in an embodiment of the present invention;

[0048] Figure 2 This is one of the methods for creating a hierarchical nested clustering tree provided in an embodiment of the present invention;

[0049] Figure 3 This is a second method for creating a hierarchical nested clustering tree provided in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of a log pattern recognition system provided in an embodiment of the present invention;

[0051] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0052] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0053] This invention discloses a log pattern recognition method, see [link to relevant documentation]. Figure 1 ,include:

[0054] S11. Segment each piece of log data collected into words, and label the word attributes of each segmented word to obtain a word segmentation part-of-speech list for each piece of log data. The word segmentation part-of-speech list includes the word corresponding to each piece of log data and the word attributes corresponding to each word.

[0055] After word segmentation, the jieba tool is needed to annotate the word attributes (including verbs, nouns, adjectives, adverbs, numerals, measure words, and pronouns) to obtain the word segmentation of each log data and the word attributes corresponding to each word segmentation, forming a word segmentation part-of-speech list.

[0056] S12. Remove the word segments that match the pre-stored words and their attributes in the common vocabulary database from the word segmentation part-of-speech list of each log data, and form the word segmentation list of each log data by combining the remaining word segments in each log data.

[0057] Word segmentation that matches both the pre-stored words and their attributes in the common vocabulary database refers to word segmentation that matches both the pre-stored words and their attributes in the common vocabulary database. The matched word segments are removed, and the word attribute annotations can also be removed simultaneously. The remaining segmented text is used as the word segmentation list for this log entry.

[0058] The pre-stored words in the commonly used vocabulary database contain any or a combination of the following pre-stored word attributes:

[0059] The general category of operational verbs, such as: process, control, operate;

[0060] Non-operational word attributes (nouns, pronouns, adjectives) in general terms, such as: system, service, equipment, device, this time, capability, dedicated;

[0061] Commonly used terms related to the source device of the log data. For example, if the log source device is an alarm device, the relevant commonly used terms are: alarm, warning, alert, etc.

[0062] The database of frequently used terms was established based on accumulated operational experience and troubleshooting issues. When new faults or complaints arise, the database is updated based on the new operational experience and troubleshooting issues.

[0063] S13. Based on the word segmentation list of each log data, vectorize each log data and cluster the vectorized log data.

[0064] The process of vectorizing each log data entry based on a word segmentation list includes:

[0065] The word segmentation list is used to select words whose total occurrence frequency in the log data exceeds a preset threshold. Based on the selected word segmentation list for each log data entry, each log data entry is vectorized using TF-IDF. In this embodiment of the invention, the total occurrence frequency of word t is first calculated, and words with an occurrence frequency greater than N are selected to calculate their TF-IDF values. N is controlled by a parameter, thus implementing a method for dynamically calculating TF-IDF.

[0066] For example, based on a word segmentation list of m log data, each log data can be vectorized using TF-IDF. Each log data can be obtained as an n-dimensional vector value, where n represents the number of all distinct words in the word segmentation list of all m log data. Vectorizing the m log data results in an m*n matrix.

[0067] TF-IDF (Term Frequency-Inverse Document Frequency) is a statistical method used to evaluate the importance of a word within a document collection or corpus. In this invention, it is used to measure the importance of different words in a single log entry and to vectorize the log text data using the metric results. TF-IDF consists of two parts: TF (Term Frequency) and IDF (Inverse Document Frequency), and the result is TF*IDF. The calculation formula is:

[0068]

[0069] Where ni,j is the number of times the word ti appears in document dj, and the denominator is the sum of the number of times all words appear in document dj.

[0070]

[0071] Where |D| represents the total number of documents in the corpus;

[0072] |{j:t i ∈d j}|: Contains the word t i Number of files (i.e., n) i,j (Number of files ≠ 0) If the word is not in the corpus, the divisor will be zero. Therefore, 1+|{j:t is generally used. i ∈d j Then:

[0073] tfidf i,j =tf i,j ×idf i

[0074] In other words, in the m*n matrix obtained in this step, the value in the i-th row and j-th column is tfidf. i,j =tf i,j ×idf i ,in, n i,j ∑ is the number of times the word corresponding to column j appears in the log of row i. k n k，j It is the sum of the occurrence counts of all word segments in m log data; |D| represents m log entries, |{j:t i ∈d j| represents the number of log entries in the m log entries corresponding to the word in the j-th column of the matrix. For example, the word "call" appears 3 times in three log entries, which is |{j:t i ∈d j}|=3.

[0075] S14. Based on the clustering results of each log data, identify the log pattern of each type of log data.

[0076] The clustering of the vectorized log data includes:

[0077] Using a hierarchical clustering method, a hierarchical nested clustering tree is created by calculating the similarity between data points of different categories in each vectorized log data. In the nested clustering tree, data points of different categories are the lowest level of the nested clustering tree, and the top level of the nested clustering tree is the root node of a cluster.

[0078] Hierarchical clustering is a type of clustering algorithm that creates a hierarchical, nested clustering tree by calculating the similarity between data points of different categories. In the clustering tree, the original data points of different categories form the lowest level, and the top level is the root node of a cluster. There are two methods for creating clustering trees: bottom-up merging and top-down splitting.

[0079] 1. Splitting method:

[0080] The splitting algorithm refers to initially grouping all samples into one cluster, and then gradually splitting them according to certain criteria until a certain condition or a predetermined number of categories is reached. Algorithm description:

[0081] Input: Sample set D, number of clusters, or a condition (usually a threshold for sample distance, in which case the number of clusters does not need to be set).

[0082] Output: Clustering results

[0083] 1. Group all samples in the sample set into one cluster;

[0084] Repeat the following steps:

[0085] 2. Within the same cluster (denoted as c), calculate the distance between any two samples and find the two samples a and b that are furthest apart;

[0086] 3. Assign samples a and b to different clusters c1 and c2;

[0087] 4. Calculate the distances between the remaining sample points in the original cluster (c) and a and b. If dis(a) < dis(b), then assign the sample point to c1; otherwise, assign it to c2. The dist calculation method mainly uses the Euclidean distance algorithm, which is as follows:

[0088]

[0089] This continues until the required number of clusters is reached or the set conditions are met.

[0090] 2. Coagulation method:

[0091] Agglomerative merging refers to initially treating each sample point as a cluster, so the size of the original clusters equals the number of sample points. These initial clusters are then merged according to certain criteria until a certain condition or a predetermined number of classifications is reached. Algorithm description:

[0092] Input: Sample set D, number of clusters, or a condition (usually a threshold for sample distance, in which case the number of clusters does not need to be set).

[0093] Output: Clustering results

[0094] 1. Treat all sample points in the sample set as an independent cluster;

[0095] Repeat steps 2-3:

[0096] 2. Calculate the distance between each pair of clusters, and find the two clusters c1 and c2 with the smallest distance;

[0097] 3. Merge clusters c1 and c2 into one cluster;

[0098] This continues until the required number of clusters is reached or the set conditions are met.

[0099] According to the log pattern recognition method provided by the present invention, the step of segmenting the collected log data into words includes:

[0100] Perform language-level judgment on each line of log data;

[0101] Log data identified as being in English will be segmented using spaces as delimiters;

[0102] Log data identified as being in Chinese and / or a mixture of Chinese and English will be segmented using the jieba word segmentation tool.

[0103] Jieba is a word segmentation plugin that can segment Chinese and / or mixed Chinese and English text into words. After word segmentation, the jieba tool is needed to classify the attributes of the segmented words (including: verbs, nouns, adjectives, adverbs, numerals, measure words, and pronouns).

[0104] Using the optimized TF-IDF algorithm, each segmented log entry is vectorized, resulting in a list of vectorized matrices. Each row represents the vectorized data of one log entry; for example, the first row is [W11, W12, ..., W1n], where n represents the number of distinct words in all log entries. Each dimension represents the vectorized data of a particular word in that log entry, ensuring each log entry has the same dimensionality. After vectorization, an m*n matrix is ​​obtained, where m represents the number of log entries and n represents the number of distinct words. A clustering algorithm is then used to cluster the vector matrix. After model training and selection, the optimal number of clusters K is chosen first. This K value is then used to retrain the final clustering model to cluster the vectorized matrix, yielding the clustering results. For each log entry in each cluster, common words are extracted, and different words are replaced with * (asterisks). The * part represents variable log information, and the common part represents the desired pattern. This process completes the log pattern recognition for each cluster.

[0105] See Figure 2 According to the log pattern recognition method provided by the present invention, creating a hierarchical nested clustering tree includes:

[0106] S21. Group all the vectorized log data into a first primitive class cluster;

[0107] S22. According to the preset splitting rules, split the first original cluster to create a first nested clustering tree;

[0108] S23. Determine whether the first nested clustering tree meets the preset conditions. If yes, execute S24: treat the first nested clustering tree as a hierarchical nested clustering tree. If no, return to S22.

[0109] The step of splitting the first original cluster according to the preset splitting rules includes:

[0110] Within the same primary class cluster, obtain the distance between every two data points, and find the first and second data points with the greatest distance;

[0111] The first data point and the second data point are assigned to different first-class clusters and second-class clusters, respectively;

[0112] Obtain the remaining data points in the first original cluster and the first and second distances between the first and second data points. If the first distance is less than the second distance, assign the remaining data points to the first cluster; otherwise, assign the remaining data points to the second cluster.

[0113] See Figure 3According to the log pattern recognition method provided by the present invention, creating a hierarchical nested clustering tree includes:

[0114] S31. Treat each data point in all vectorized log data as a second primitive cluster;

[0115] S32. Merge the second original clusters according to the preset merging rules to create a second nested clustering tree;

[0116] S33. Determine whether the second nested clustering tree meets the preset conditions. If yes, execute S34: treat the second nested clustering tree as a hierarchical nested clustering tree. If no, return to S32.

[0117] The merging of the second original cluster according to the preset merging rules includes:

[0118] Calculate the distance between every two second primitive clusters, and find the two second primitive clusters with the smallest distance;

[0119] Merge the two second primitive clusters with the smallest distance into one cluster.

[0120] To further understand the method of this embodiment, in a specific example:

[0121] The following table shows the collected system log data (10 log samples are displayed here):

[0122] Table 1

[0123]

[0124]

[0125] Each line represents a log entry, and the data shows that these are log entries with a mix of Chinese and English.

[0126] First, each line of data is analyzed at the language level to determine whether it is an English or Chinese log. If it is an English log, it is segmented using spaces. If it is a Chinese log, it is segmented using the jieba segmentation tool (logs with mixed Chinese and English are uniformly classified as Chinese logs; with certain processing methods, only the Chinese part can be segmented using jieba, while the English part is segmented using " / "). Meaningless words of length 1 are filtered out, such as words like "I", "and", etc. This yields a segmented list for each log entry [word1, word2, ..., wordm]. Since different logs have different content, the length of each segmented log entry will vary.

[0127] Then, based on the established common vocabulary database corresponding to the log source system, words from the common vocabulary database are removed from the word segmentation list of each log data.

[0128] For example, the word segmentation list for the second log data could be: this time (pronoun), REMOTE, call (verb), total duration (noun), 997 (numeral), ms (quantifier). The pronouns and nouns in the word segmentation list can be entered into the generic word (noun, pronoun, adjective) library in the common vocabulary database for matching, and the verbs can be entered into the generic operation word (verb) library in the common vocabulary database for matching. If "this time" and "call" match successfully, then these two words are removed from the word segmentation list of this data, and the word attribute annotations are also removed. Thus, the word segmentation list for the second log data would be: REMOTE, total duration, 997, ms.

[0129] Because the same words can have different parts of speech in different contexts—for example, words like "reading," "input," "homework," and "format" can function as both verbs and nouns—after determining the word attributes of the segmented text based on the contextual relationships in the log data using the jieba tool, further matching these word attributes with different types of pre-stored words in a common vocabulary database is performed. This removes words with weaker influence but high frequency, thus improving the accuracy of log pattern recognition.

[0130] Based on the word segmentation results, the optimized TF-IDF (Term Frequency-Inverse Document Frequency) algorithm is used to transform the segmented data into vectorized data. This results in each log entry having an n-dimensional vector value, where n represents the number of distinct words in the word segmentation list of all m log entries. The processed m log entries form an m*n matrix, as shown in Table 2-4 below. Table 2 represents the original logs, Table 3 represents the word segmentation results, and Table 4 represents the resulting m*n matrix.

[0131] Table 2

[0132]

[0133] Table 3

[0134]

[0135] Table 4

[0136]

[0137]

[0138] Then, a hierarchical clustering algorithm is used to cluster the m vectorized log data;

[0139] Using the optimized TF-IDF algorithm, each segmented log entry is vectorized, resulting in a list of vectorized matrices. Each row represents the vectorized data of one log entry; for example, the first row is [W11, W12, ..., W1n], where n represents the number of distinct words in all log entries. Each dimension represents the vectorized data of a particular word in that log entry, ensuring each log entry has the same dimensionality. After vectorization, an m*n matrix is ​​obtained, where m represents the number of log entries and n represents the number of distinct words. A clustering algorithm is then used to cluster the vector matrix. After model training and selection, the optimal number of categories K is chosen first. This K value is then used to retrain the final clustering model to cluster the vectorized matrix, yielding the clustering results. For each log entry in each category, common words are extracted, and different words are replaced with * (asterisks). The * represents variable log information, and the common part is the pattern we need to extract. This process completes the log pattern recognition for each category. The log pattern recognition output of this system is shown in Table 5 below.

[0140]

[0141]

[0142] The raw data shows eight log output types. Looking at the final log pattern, it's clear that even a small number of abnormal logs can be accurately identified as different log categories, thus simultaneously serving as a log anomaly detection function. This method leverages the semantic similarity of log data to group logs with inconsistent wording but similar meanings into one category and output them accurately, without relying on external input labels.

[0143] This invention performs word segmentation on the collected log data, annotates the word attributes of each segmented word to obtain a word segmentation part-of-speech list for each log data entry, and removes common and generalized words from the log data through word attribute correspondence, which can significantly improve processing efficiency and the accuracy of quantified log data. Based on the word segmentation list of each log data entry, each log data entry is vectorized, and the vectorized log data entries are clustered, eliminating the need for manual data labeling. By using word segmentation tools and optimized TF-IDF, unstructured text data is transformed into vector data, quantifying the log text data and reflecting the semantic information of the logs, enabling a better understanding of the log data and identification of log patterns for each type of log data.

[0144] This invention discloses a log pattern recognition system, see [link to documentation]. Figure 4 ,include:

[0145] The word segmentation part-of-speech list determination module 10 is used to segment the collected log data into words, mark the word attributes of each segmented word, and obtain the word segmentation part-of-speech list of each log data. The word segmentation part-of-speech list includes the word corresponding to each log data and the word attributes corresponding to each segmented word.

[0146] The word segmentation list determination module 20 is used to remove word segments that match the pre-stored words and attributes in the common vocabulary database from the word segmentation part-of-speech list of each log data, and form the word segmentation list of each log data by the remaining word segments in each log data.

[0147] Clustering module 30 is used to vectorize each log data based on the word segmentation list of each log data, and then cluster the vectorized log data.

[0148] The log pattern recognition module 40 is used to identify the log pattern of each type of log data based on the results of clustering each log data.

[0149] Furthermore, the word segmentation part-of-speech list determination module 10 is used for:

[0150] Perform language-level judgment on each line of log data;

[0151] Log data identified as being in English will be segmented using spaces as delimiters;

[0152] Log data identified as being in Chinese and / or a mixture of Chinese and English will be segmented using the jieba word segmentation tool.

[0153] Preferably, the pre-stored words in the commonly used vocabulary database include any or a combination of the following pre-stored word attributes:

[0154] Generic operational term attributes;

[0155] Non-operational word attributes of generic categories;

[0156] Commonly used terminology attributes related to the source device of the log data.

[0157] The clustering module 30 is further configured to include:

[0158] Based on a word segmentation list of m log data, each log data is vectorized using TF-IDF. Each log data can be obtained as an n-dimensional vector value, where n represents the number of all distinct words in the word segmentation list of all m log data. Vectorizing the m log data results in an m*n matrix, where the value in the i-th row and j-th column is the TF-IDF value. i,j =tf i,j ×idf i ,in, ni,j ∑ is the number of times the word corresponding to column j appears in the log of row i. k n k,j It is the sum of the occurrence counts of all word segments in m log data; |D| represents m log entries, |{j:t i ∈d j}| represents the number of log entries in which the word corresponding to the j-th column of the matrix appears in m log entries.

[0159] The clustering module 30 is further configured to include:

[0160] Using a hierarchical clustering method, a hierarchical nested clustering tree is created by calculating the similarity between data points of different categories in each vectorized log data. In the nested clustering tree, data points of different categories are the lowest level of the nested clustering tree, and the top level of the nested clustering tree is the root node of a cluster.

[0161] The above process of creating a hierarchical nested clustering tree includes:

[0162] S21. Group all the vectorized log data into a first primitive class cluster;

[0163] S22. According to the preset splitting rules, split the first original cluster to create a first nested clustering tree;

[0164] S23. Determine whether the first nested clustering tree meets the preset conditions. If yes, then treat the first nested clustering tree as a hierarchical nested clustering tree. If no, return to S22.

[0165] The step of splitting the first original cluster according to the preset splitting rules includes:

[0166] Within the same primary class cluster, obtain the distance between every two data points, and find the first and second data points with the greatest distance;

[0167] The first data point and the second data point are assigned to different first-class clusters and second-class clusters, respectively;

[0168] Obtain the remaining data points in the first original cluster and the first and second distances between the first and second data points. If the first distance is less than the second distance, assign the remaining data points to the first cluster; otherwise, assign the remaining data points to the second cluster.

[0169] Furthermore, creating a hierarchical nested clustering tree includes:

[0170] S31. Treat each data point in all vectorized log data as a second primitive cluster;

[0171] S32. Merge the second original clusters according to the preset merging rules to create a second nested clustering tree;

[0172] S33. Determine whether the second nested clustering tree meets the preset conditions. If yes, then treat the second nested clustering tree as a hierarchical nested clustering tree. If no, return to S32.

[0173] The merging of the second original cluster according to the preset merging rules includes:

[0174] Calculate the distance between every two second primitive clusters, and find the two second primitive clusters with the smallest distance;

[0175] Merge the two second primitive clusters with the smallest distance into one cluster.

[0176] Figure 5 A schematic diagram of the physical structure of an electronic device is provided. This electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540. The processor 510, communication interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can invoke logical instructions from the memory 530 to execute a log pattern recognition method, which includes:

[0177] S1. Segment each piece of log data collected into words, and label the word attributes of each segmented word to obtain a word segmentation part-of-speech list for each piece of log data. The word segmentation part-of-speech list includes the word corresponding to each piece of log data and the word attributes corresponding to each word.

[0178] S2. Remove the word segments that match the pre-stored words and their attributes in the common vocabulary database from the word segmentation part-of-speech list of each log data, and form the word segmentation list of each log data with the remaining word segments.

[0179] S3. Based on the word segmentation list of each log data, vectorize each log data and cluster the vectorized log data.

[0180] S4. Based on the clustering results of each log data, identify the log pattern of each type of log data.

[0181] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0182] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute a log pattern recognition method, the method comprising:

[0183] S1. Segment each piece of log data collected into words, and label the word attributes of each segmented word to obtain a word segmentation part-of-speech list for each piece of log data. The word segmentation part-of-speech list includes the word corresponding to each piece of log data and the word attributes corresponding to each word.

[0184] S2. Remove the word segments that match the pre-stored words and their attributes in the common vocabulary database from the word segmentation part-of-speech list of each log data, and form the word segmentation list of each log data with the remaining word segments.

[0185] S3. Based on the word segmentation list of each log data, vectorize each log data and cluster the vectorized log data.

[0186] S4. Based on the clustering results of each log data, identify the log pattern of each type of log data.

[0187] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform a log pattern recognition method, the method comprising:

[0188] S1. Segment each piece of log data collected into words, and label the word attributes of each segmented word to obtain a word segmentation part-of-speech list for each piece of log data. The word segmentation part-of-speech list includes the word corresponding to each piece of log data and the word attributes corresponding to each word.

[0189] S2. Remove the word segments that match the pre-stored words and their attributes in the common vocabulary database from the word segmentation part-of-speech list of each log data, and form the word segmentation list of each log data with the remaining word segments.

[0190] S3. Based on the word segmentation list of each log data, vectorize each log data and cluster the vectorized log data.

[0191] S4. Based on the clustering results of each log data, identify the log pattern of each type of log data.

[0192] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0193] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A log pattern recognition method, characterized in that: include: Segment each piece of collected log data, mark the word attributes of each segmented word after segmentation, and obtain a segmented word part-of-speech list for each piece of log data, wherein the segmented word part-of-speech list includes the segmented word corresponding to each piece of log data and the word attribute corresponding to each segmented word; Remove the segmented words that match the pre-stored words and pre-stored word attributes in the common vocabulary database from the segmented word part-of-speech list of each log data, and form the segmented word list of each log data with the remaining segmented words; Based on the word list of each log data, each log data is vectorized and clustered; Based on the clustering results of each log data, the log pattern of each type of log data is identified; The clustering of each vectorized log data includes: By using a hierarchical clustering method, a hierarchical nested clustering tree is created by calculating the similarity between different category data points in each vectorized log data; in the nested clustering tree, the data points of different categories are the lowest level of the nested clustering tree, and the top level of the nested clustering tree is the root node of a cluster; The step of creating a hierarchical nested clustering tree includes: S21, classifying all vectorized log data into a first original class cluster; S22, splitting the first original cluster according to a preset splitting rule to create a first nested clustering tree; S23, judging whether the first nested clustering tree meets a preset condition, if so, taking the first nested clustering tree as a hierarchical nested clustering tree, if not, returning to S22; The step of splitting the first original cluster according to a preset splitting rule includes: Get the distance between every two data points in the same first original cluster, and find the first data point and the second data point with the farthest distance; Assign the first data point and the second data point to different first-type clusters and second-type clusters respectively; Obtain the first distance and the second distance between the remaining data points in the first original cluster and the first data point and the second data point. If the first distance is less than the second distance, the other data points are assigned to the first cluster; otherwise, the other data points are assigned to the second cluster. Alternatively, the step of creating a hierarchical nested clustering tree includes: S31, taking each data point in all the vectorized log data as a second original cluster; S32, merging the second original clusters according to a preset merging rule to create a second nested clustering tree; S33, judging whether the second nested clustering tree meets a preset condition, if so, taking the second nested clustering tree as a hierarchical nested clustering tree, if not, returning to S32; The merging of the second original class clusters according to the preset merging rule includes: Calculate the distance between every two second-class clusters and find the two second-class clusters with the smallest distance; Merge the two second original clusters with the smallest distance into one cluster.

2. The log pattern recognition method according to claim 1, characterized in that: The word segmentation of the collected log data includes: Make language-level judgments for each line of log data; The log data judged to be in English is segmented by spaces; The log data judged to be in Chinese and / or a mixture of Chinese and English are segmented using the Jieba word segmentation tool.

3. The log pattern recognition method according to claim 1, characterized in that: The pre-stored words in the common vocabulary database include any one or a combination of the following pre-stored word attributes: The general class of operational terms attributes; The non-operational word attributes of the general class; Common word attributes related to the source device of the log data.

4. The log pattern recognition method according to claim 1, characterized in that: The vectorization of each log data based on the word segmentation list of each log data includes: Filter out the words whose total number of occurrences in the log data is greater than the preset threshold from the word list. Based on the filtered word list of each log data, use the TF-IDF method to vectorize each log data.

5. A log pattern recognition system, characterized in that: include: A segmentation part-of-speech list determination module is used to segment the collected log data, mark the word attributes of each segmented word after segmentation, and obtain a segmentation part-of-speech list for each log data, wherein the segmentation part-of-speech list includes the segmentation corresponding to each log data and the word attributes corresponding to each segmentation; A word segmentation list determination module is used to remove the word segments that match the pre-stored words and pre-stored word attributes in the common vocabulary database from the word segmentation part-of-speech list of each log data, and form the word segmentation list of each log data with the remaining word segments in each log data; The clustering module is used to vectorize each log data based on the word list of each log data, and cluster each vectorized log data; A log pattern recognition module is used to identify the log pattern of each type of log data based on the results of clustering each log data; The clustering module is also used to: By using a hierarchical clustering method, a hierarchical nested clustering tree is created by calculating the similarity between different category data points in each vectorized log data; in the nested clustering tree, the data points of different categories are the lowest level of the nested clustering tree, and the top level of the nested clustering tree is the root node of a cluster; The step of creating a hierarchical nested clustering tree includes: S21, classifying all vectorized log data into a first original class cluster; S22, splitting the first original cluster according to a preset splitting rule to create a first nested clustering tree; S23, judging whether the first nested clustering tree meets a preset condition, if so, taking the first nested clustering tree as a hierarchical nested clustering tree, if not, returning to S22; The step of splitting the first original cluster according to a preset splitting rule includes: Get the distance between every two data points in the same first original cluster, and find the first data point and the second data point with the farthest distance; Assign the first data point and the second data point to different first-type clusters and second-type clusters respectively; Obtain the first distance and the second distance between the remaining data points in the first original cluster and the first data point and the second data point. If the first distance is less than the second distance, the other data points are assigned to the first cluster; otherwise, the other data points are assigned to the second cluster. Alternatively, the step of creating a hierarchical nested clustering tree includes: S31, taking each data point in all the vectorized log data as a second original cluster; S32, merging the second original clusters according to a preset merging rule to create a second nested clustering tree; S33, judging whether the second nested clustering tree meets a preset condition, if so, taking the second nested clustering tree as a hierarchical nested clustering tree, if not, returning to S32; The merging of the second original class clusters according to the preset merging rule includes: Calculate the distance between every two second-class clusters and find the two second-class clusters with the smallest distance; Merge the two second original clusters with the smallest distance into one cluster.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the log pattern recognition method according to any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the log pattern recognition method according to any one of claims 1 to 4 are implemented.

Images