Log clustering method and device based on longest common subsequence, and storage medium

Abstract

This invention discloses a log clustering method, device, and storage medium based on the longest common subsequence (LCS), relating to the field of log analysis technology. The method includes: reading multi-source logs; preprocessing the multi-source logs; constructing a fixed-depth prefix tree; clustering the preprocessed multi-source logs based on each log sequence in the preprocessed multi-source logs using the fixed-depth prefix tree and the LCS; determining the storage dimension of the clustering results for the preprocessed multi-source logs; and storing the multi-source logs and the clustering results. This invention improves the efficiency and accuracy of online log clustering algorithms, meeting the urgent need for online clustering of system logs in practical scenarios.

Description

Technical Field

[0001] This invention relates to the field of log analysis technology, specifically to a log clustering method, device, and storage medium based on the longest common subsequence. Background Technology

[0002] As the complexity of modern computer systems continues to increase, their deployment and management become increasingly difficult. Real-time monitoring of system operation status and timely detection and resolution of system anomalies have become core requirements in many scenarios. Logs, as a core data resource for recording system operation status, contain key information such as timestamps, event levels, and operation descriptions, and are widely used in various tasks such as anomaly detection, fault diagnosis, performance optimization, and user behavior analysis. The data mining models relied upon for these log analysis tasks typically require structured input, but raw logs are unstructured data in text format. Therefore, log parsing becomes the first and crucial step in log analysis.

[0003] The most commonly used method in log parsing is log clustering algorithm. Although many studies have been dedicated to the development of log clustering algorithms, existing algorithms still have the following significant limitations.

[0004] 1. Traditional log clustering relies on manual design and maintenance of regular expressions, but this approach can no longer meet real-world needs. It has two main drawbacks: First, the log volume is growing explosively, with large service systems generating logs every day, making manual processing difficult; second, predefined regular expressions require manual design by professionals and have poor versatility.

[0005] 2. Most traditional methods are designed for offline logs, requiring the collection of all logs within a specific period before model training, which cannot meet the needs of online real-time processing.

[0006] 3. A few methods with online log clustering capabilities (such as SHISO and Drain) can achieve online streaming log processing, but they perform poorly in terms of clustering accuracy. Moreover, the clustering accuracy is further reduced when dealing with logs containing Chinese characters, making it difficult to meet the needs of modern log clustering.

[0007] In summary, with the continuous growth of log data volume and the increasing requirements for real-time log clustering, the shortcomings of existing log clustering methods in balancing accuracy and efficiency are becoming increasingly apparent. There is an urgent need to develop an efficient and accurate online log clustering algorithm to meet the pressing needs of online system log clustering in real-world scenarios. Summary of the Invention

[0008] This invention addresses the problems of existing log clustering methods in balancing accuracy and efficiency by providing a log clustering method, device, and storage medium based on the longest common subsequence. This improves the efficiency and accuracy of online log clustering algorithms and meets the urgent need for online clustering of system logs in real-world scenarios.

[0009] The present invention is achieved through the following technical solution.

[0010] Firstly, a log clustering method based on the longest common subsequence is provided, the method comprising:

[0011] Read logs from multiple sources;

[0012] The multi-source logs are preprocessed;

[0013] A fixed-depth prefix tree is constructed, and based on each log sequence in the preprocessed multi-source logs, the preprocessed multi-source logs are clustered using the fixed-depth prefix tree and the longest common subsequence (LCS).

[0014] Determine the storage dimension for the clustering results of the preprocessed multi-source logs, and store the multi-source logs and clustering results.

[0015] In some embodiments, preprocessing of the multi-source logs includes:

[0016] Replace the dynamic variables in the multi-source logs with the first normalized wildcard.

[0017] In some embodiments, preprocessing the multi-source logs further includes:

[0018] The Chinese content in the multi-source logs is segmented into words.

[0019] After word segmentation, the multi-source logs are divided into standardized tokens based on spaces.

[0020] In some embodiments, constructing a fixed-depth prefix tree includes:

[0021] A variable token filtering mechanism is set for the prefix tree so that, during the matching phase, the variables contained in the multi-source logs are matched with a second preset wildcard.

[0022] Set an upper limit on the number of child nodes for each node in the prefix tree.

[0023] In some embodiments, based on each log sequence in the preprocessed multi-source logs, clustering is performed on the preprocessed multi-source logs using the fixed-depth prefix tree and the longest common subsequence, including:

[0024] Determine the LCS of each log sequence in the preprocessed multi-source logs with each log template sequence in the log group under each leaf node in the prefix tree;

[0025] The length of the LCS is calculated as the ratio of the length of the longer sequence among the corresponding log sequence in each log sequence of the multi-source logs and the length of each log template sequence, and is used as the similarity score.

[0026] ;

[0027] in, , representing the sequence of logs to be matched in a multi-source log; , indicating a log template sequence, , Sequences Length; This represents the maximum length of two sequences; Indicates the length of the LCS;

[0028] Based on the ratio, target log groups with the highest similarity value to each log sequence are selected;

[0029] The highest similarity value is compared with a predefined threshold. If the highest similarity value is greater than the predefined threshold, the target log group is returned as the best matching result; if the highest similarity value is less than or equal to the predefined threshold, it indicates that no suitable log group was found.

[0030] In some embodiments, if no log group exists under any leaf node in the prefix tree or if no matching log group is found, a new log group identifier is generated according to preset rules, a new log group is created using the current log as the initial template, and the prefix tree is updated.

[0031] The process includes generating a new log group identifier based on preset rules, creating a new log group with the current log as the initial template, and updating the prefix tree. This includes using the preset first N tokens obtained after processing the current log with preset word segmentation rules as internal nodes of the prefix tree, and storing the template of the current log and the current log identifier ID as associated data in the leaf nodes of the path.

[0032] In some embodiments, the method further includes: after returning the target log group as the best matching result when the highest similarity value is greater than the predefined threshold, updating the prefix tree.

[0033] Updating the prefix tree includes: adding the current log ID to the associated ID list of the corresponding log group, and updating the log template.

[0034] The update log template includes: comparing the token sequence of the current log with the token sequence of the original template of the log group, solving the LCS of the token sequence of the current log with the token sequence of the original template of the log group through a dynamic programming algorithm, and using the LCS as a benchmark, replacing the part of the longer sequence in the token sequence of the current log with the token sequence of the original template of the log group that does not match the LCS with a preset wildcard.

[0035] Secondly, a log clustering device based on the longest common subsequence is provided, the device comprising:

[0036] The log reading module is used to read logs from multiple sources.

[0037] The log preprocessing module is used to: preprocess the multi-source logs;

[0038] The log clustering module is used to: construct a fixed-depth prefix tree, and cluster the preprocessed multi-source logs based on each log sequence in the preprocessed multi-source logs using the fixed-depth prefix tree and the longest common subsequence (LCS).

[0039] The storage module is used to: determine the storage dimension of the clustering results of the preprocessed multi-source logs, and store the multi-source logs and the clustering results.

[0040] Thirdly, a log clustering device based on the longest common subsequence is provided, the device comprising:

[0041] At least one processor;

[0042] At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions implementing the method described in any of the above when executed by the at least one processor.

[0043] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform the method described in any of the preceding embodiments.

[0044] Compared with existing technologies, this invention has the following advantages and beneficial effects: It uses the ratio of the length of the longest common subsequence to that of the longer sequence as the similarity between two log sequences, and clusters logs with high similarity into one class, significantly improving clustering accuracy compared to traditional methods. Simultaneously, addressing the problem of poor clustering performance for logs containing Chinese characters in existing log clustering algorithms, a Chinese word segmentation processing mechanism is introduced. By optimizing the similarity calculation of Chinese logs, clustering accuracy is further improved. To optimize algorithm performance, a fixed-depth tree structure is introduced. Log data is first routed to the leaf nodes of the tree via the tree structure's routing rules. Each leaf node contains several log groups (or creates a new log group based on newly routed logs if none exist). Then, the aforementioned similarity algorithm is applied between groups to complete clustering, thereby significantly improving the efficiency of clustering operations. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart of a log clustering method based on the longest common subsequence according to an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of log clustering according to an embodiment of the present invention.

[0048] Figure 3 This is a structural block diagram of a log clustering device based on the longest common subsequence according to an embodiment of the present invention.

[0049] Figure 4 This is a schematic diagram of a log clustering device based on the longest common subsequence according to an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this invention are only for explaining this invention and are not intended to limit this invention.

[0051] To facilitate understanding of the technical solution of this invention, some terms involved in this invention will be explained below.

[0052] Algorithm: A set of clear, defined steps or rules for solving a problem. In computer science, an algorithm is a finite sequence of steps to perform a computational or data processing task. Algorithms can be written as computer programs, enabling computers to perform specific tasks.

[0053] Log clustering is a technique that uses unsupervised machine learning to automatically group massive amounts of unstructured log data into a limited number of categories based on the similarity of their content, patterns, or contextual features.

[0054] Prefix tree: A multi-way tree data structure that saves space by sharing string prefixes and supports efficient string storage and retrieval.

[0055] Longest common subsequence: A sequence is a subsequence of two or more known sequences, and is the longest of all sequences that meet this condition.

[0056] On the one hand, the present invention provides a log clustering method based on the longest common subsequence. Figure 1 This is a flowchart of a log clustering method based on the longest common subsequence according to an embodiment of the present invention. (Reference) Figure 1 The log clustering method based on the longest common subsequence includes S10 to S40.

[0057] In S10, multi-source logs are acquired and read.

[0058] Specifically, for Kafka message queue data sources, real-time subscription and retrieval of log streams are achieved through consumer group management, partition balancing, and abnormal reconnection mechanisms, ensuring stable input of streaming logs.

[0059] For local log file data sources, it provides file path configuration and read mode selection (full read at once, batch read, etc.), and parses log data line by line and filters blank lines and invalid characters.

[0060] For command-line input data sources, listen to the standard input stream, receive log data manually entered by the user, and perform real-time reading.

[0061] In S20, multi-source logs are preprocessed.

[0062] In some embodiments, preprocessing of multi-source logs includes replacing dynamic variables in the multi-source logs with a first standardized wildcard. Specifically, to address the problem of variables interfering with clustering determination in logs, a variable replacement strategy is adopted to uniformly replace all dynamic variables with standardized wildcards, eliminating the impact of variable differences on the essential characteristics of the logs from the root. For example, to meet the diverse log preprocessing needs of business scenarios, users can configure simple regular expressions for high-frequency business-irrelevant variables such as IP (Internet Protocol) addresses, business IDs, and random numbers based on domain expert knowledge. An algorithm is used to automatically call these preset regular expressions in the log processing pipeline, accurately identifying target variables in the original logs through regular expression matching, completing automated filtering and replacement, removing redundant variable information, and preserving the core semantic structure. For example, the input log "The timestamp obtained is: 1766028633000" will, after dynamic variable replacement, be standardized to "The timestamp obtained is: <:NUM:>".

[0063] In some embodiments, preprocessing of multi-source logs further includes: segmenting the Chinese content in the multi-source logs into words; and dividing the segmented multi-source logs into standardized tokens based on spaces. Specifically, addressing the problem of low feature distinguishability caused by existing methods using spaces to separate Chinese logs, this invention first segments the Chinese content in the logs into words, and then divides the entire log segment into standardized tokens based on spaces, thereby improving the feature distinguishability of Chinese logs and preparing for subsequent log clustering tasks.

[0064] In S30, a fixed-depth prefix tree is constructed. Based on the log sequences in the preprocessed multi-source logs, the preprocessed multi-source logs are clustered using the fixed-depth prefix tree and the longest common subsequence.

[0065] Log clustering includes: constructing a fixed-depth prefix tree, classifying logs based on the longest common subsequence, and updating the prefix tree. In some embodiments, constructing the fixed-depth prefix tree includes: setting a variable token filtering mechanism for the prefix tree to match a second preset wildcard for variables contained in multi-source logs during the matching phase; and setting an upper limit on the number of child nodes for each node in the prefix tree.

[0066] refer to Figure 2The log clustering strategy is based on matching the prefix tree structure with the token at the start position of the log message. For a segmented log sequence, starting from the root node of the prefix tree, the child nodes corresponding to the first and subsequent tokens in the log sequence are matched sequentially until the leaf node of the prefix tree is reached. Finally, the leaf node is returned as the matching result. To further ensure the rationality of the prefix tree structure and the efficiency of token matching, two optimization measures are designed in this invention: a variable token filtering mechanism and a child node number limitation mechanism.

[0067] Variable Token Filtering Mechanism: During the log matching phase, only tokens without variables are selected for calculation. If a log message uses a parameter containing a variable as a token, the variable is directly matched to a predefined special internal node "*", avoiding prefix tree branch explosion caused by the diversity of parameter values. For example, for the input log: "Member 0180 Login Successful", after dynamic variable replacement and Chinese word segmentation, it becomes "Member <:NUM:> Login Successful" (corresponding to the first normalized wildcard in preprocessing), where the first token "Member" is matched. Figure 2 The first child node is "Member", and the second token "<:NUM:>" will match the child node "*" (the second preset wildcard) of the node "Member".

[0068] Child node count limit (fixed depth) mechanism: By setting a maximum child node parameter, the upper limit of the number of child nodes of a single node in the prefix tree is constrained. When the number of child nodes of a node reaches the threshold, all subsequent tokens that cannot match the existing child nodes of that node will be matched with the special internal node "*" under that node, thereby further controlling the size of the prefix tree and improving the overall matching efficiency. For example, for the input log: "The timestamp obtained is: 1766028633000", after dynamic variable replacement and Chinese word segmentation, it becomes "The timestamp obtained is: <:NUM:>". Since the number of child nodes in the first level exceeds the limit, the last node is "*", and the first token of the log does not match the corresponding child node, so it is automatically matched. Figure 2 If the last node in the middle is "*", and the number of nodes in the third layer does not exceed the upper limit, then a normal match will be made.

[0069] Based on the log sequences in the preprocessed multi-source logs, clustering is performed on the preprocessed multi-source logs using a fixed-depth prefix tree and the longest common subsequence, including S31 to S34.

[0070] In S31, each log sequence in the preprocessed multi-source logs is determined, and its longest common subsequence (LCS) is found among the log template sequences in the log groups under each leaf node in the prefix tree. S31 then routes the logs to a leaf node containing a list of log groups, which were added according to the internal node routing rules. Next, a log group is selected for the newly routed logs.

[0071] In S32, to normalize the similarity to the [0,1] interval, the ratio of the LCS length to the longer sequence length between the corresponding log sequence and the log template sequence in each log sequence from multiple sources is calculated as the similarity:

[0072] ;

[0073] in, , representing the sequence of logs to be matched in a multi-source log; , indicating a log template sequence, , Sequences Length, This represents the maximum length of the two sequences. The Longest Common Subsequence (LCS) is the longest subsequence that exists in both S and T and whose elements are in the same order as the original sequences (not necessarily consecutive). Its length is denoted as LCS(S,T). The closer the value is to 1, the stronger the structural or semantic relationship between the two sequences. A value of 0 indicates that the two sequences have no common subsequences, that is, the two sequences are completely different.

[0074] In S32, target log groups with the highest similarity values ​​to each log sequence are selected based on the ratio. Specifically, after calculating the similarity between logs, the target log group with the highest similarity can be selected.

[0075] In step S34, the highest similarity value is compared with a predefined threshold. If the highest similarity value is greater than the predefined threshold, the target log group is returned as the best match result; if the highest similarity value is less than or equal to the predefined threshold, it indicates that no suitable log group was found. Specifically, the highest similarity value of the group is compared with a predefined threshold. If the similarity value is higher than the threshold, the log group is returned as the best match result; if it is lower than or equal to the threshold, a preset flag is returned, indicating that no suitable log group was found.

[0076] In some embodiments, if no log group exists under any leaf node in the prefix tree or if no suitable log group is found, a new log group identifier is generated according to a preset rule, a new log group is created using the current log as the initial template, and the prefix tree is updated. Generating a new log group identifier according to a preset rule, creating a new log group using the current log as the initial template, and updating the prefix tree includes: using the preset first N tokens obtained after processing the current log according to a preset word segmentation rule as internal nodes of the prefix tree, and storing the template of the current log and the current log identifier (ID) as associated data in the leaf nodes of that path.

[0077] Specifically, if no log group exists under any leaf node in the prefix tree, or if no suitable log group is found, a unique log group identifier is generated according to the system's preset rules. A new log group is created using the current log as the initial template, and the prefix tree is updated. The specific operations are as follows: The first N tokens obtained after processing the current log using preset word segmentation rules are used as internal nodes of the prefix tree (if the path corresponding to the token sequence already exists, the node is reused; otherwise, new nodes are created in sequence); then, the template and unique ID of the current log are stored as associated data in the leaf node of that path, providing data structure support for the rapid matching and clustering of subsequent logs.

[0078] In some embodiments, the method further includes: after returning the target log group as the best matching result when the highest similarity value is greater than a predefined threshold, updating the prefix tree. Updating the prefix tree includes: adding the current log ID to the associated ID list of the corresponding log group and updating the log template. Updating the log template includes: comparing the token sequence of the current log with the token sequence of the original template of the log group, solving the LCS of the token sequence of the current log with the token sequence of the original template of the log group using a dynamic programming algorithm, and using the LCS as a benchmark, replacing the part of the longer sequence in the token sequence of the current log with the token sequence of the original template of the log group that does not match the LCS with a preset wildcard, thereby generating an updated template that is compatible with the features of new and old logs within the group.

[0079] In S40, the storage dimension of the clustering results of the preprocessed multi-source logs is determined, and the multi-source logs and clustering results are stored.

[0080] Specifically, the core storage dimensions of the clustering results are clearly defined, covering key information such as the original log text, log template (the standardized log pattern extracted after clustering), the category identifier of the log (cluster ID), and log time, ensuring the integrity of the stored content. Furthermore, it supports multiple storage methods, including relational databases (such as MySQL and GoldenDB) and file systems (such as JSON, CSV, and text files). In file storage scenarios, it supports common formats such as JSON and CSV, facilitating subsequent offline analysis and data export.

[0081] This invention proposes a method for real-time online log clustering with high accuracy. This method uses the ratio of the length of the longest common subsequence to that of the longer sequence as the similarity between two log sequences, and clusters logs with high similarity into one class. This design significantly improves clustering accuracy compared to traditional methods. Furthermore, addressing the issue of poor clustering performance for logs containing Chinese characters in existing log clustering algorithms, a Chinese word segmentation mechanism is introduced. By optimizing the similarity calculation of Chinese logs, clustering accuracy is further improved. In addition, to optimize algorithm performance, a fixed-depth tree structure is introduced. Log data is first routed to the leaf nodes of the tree according to the routing rules of the tree structure. Each leaf node contains several log groups (or creates a new log group based on newly routed logs if none exist). Then, the aforementioned similarity algorithm is applied among the groups to complete clustering, thereby significantly improving the efficiency of the clustering operation.

[0082] On the other hand, the present invention provides a log clustering device based on the longest common subsequence. Figure 3 This is a structural block diagram of a log clustering device based on the longest common subsequence according to an embodiment of the present invention. (Reference) Figure 3 The log clustering device based on the longest common subsequence includes: a log reading module, a preprocessing module, a log clustering module, and a storage module.

[0083] The log reading module is used to read logs from multiple sources.

[0084] The preprocessing module is used to preprocess multi-source logs.

[0085] The log clustering module is used to: construct a fixed-depth prefix tree, and cluster the preprocessed multi-source logs based on each log sequence in the preprocessed multi-source logs using the fixed-depth prefix tree.

[0086] The storage module is used to: determine the storage dimension of the clustering results of the preprocessed multi-source logs, and store the multi-source logs and clustering results.

[0087] For further details regarding the log clustering device based on the longest common subsequence, please refer to the previous description of the log clustering method based on the longest common subsequence, which will not be repeated here.

[0088] In implementing the functions of the integrated modules described above in hardware, this embodiment of the invention provides a structure for the log clustering device based on the longest common subsequence involved in the above embodiments. Figure 4 This is a schematic diagram of a log clustering device based on the longest common subsequence according to an embodiment of the present invention. (Reference) Figure 4 The log clustering device based on the longest common subsequence includes: at least one processor; and at least one memory. The at least one memory is coupled to the at least one processor and stores instructions for execution by the at least one processor, which, when executed by the at least one processor, implement the method described above.

[0089] A processor can be a set of logic blocks, modules, and circuits that implement or execute the various exemplary logic blocks, modules, and circuits described in connection with embodiments of the present invention. The processor can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in connection with embodiments of the present invention. A processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc.

[0090] The memory may be read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0091] In one implementation, the memory can exist independently of the processor. The memory can be connected to the processor via a bus and used to store instructions or program code. When the processor calls and executes the instructions or program code stored in the memory, it can implement the method provided in the embodiments of the present invention. In another implementation, the memory can also be integrated with the processor.

[0092] On the other hand, the present invention also provides a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) storing computer program instructions that, when executed on a computer, cause the computer to perform the method as described in any of the above embodiments.

[0093] Exemplary examples show that the aforementioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memory (EPROMs), cards, sticks, or key drives, etc.). The various computer-readable storage media described in this invention may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0094] This invention provides a computer program that, when run on a computer, causes the computer to perform the method of any of the above embodiments.

[0095] This invention provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the method of any of the above embodiments.

[0096] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for log clustering based on longest common subsequence, characterized in that, The method includes: Read logs from multiple sources; The multi-source logs are preprocessed; A fixed-depth prefix tree is constructed, and based on each log sequence in the preprocessed multi-source logs, the preprocessed multi-source logs are clustered using the fixed-depth prefix tree and the longest common subsequence (LCS). Determine the storage dimension for the clustering results of the preprocessed multi-source logs, and store the multi-source logs and clustering results.

2. The method of claim 1, wherein, Preprocessing the multi-source logs includes: Replace the dynamic variables in the multi-source logs with the first normalized wildcard.

3. The method of claim 2, wherein, Preprocessing the multi-source logs further includes: The Chinese content in the multi-source logs is segmented into words; After word segmentation, the multi-source logs are divided into standardized tokens based on spaces.

4. The method of claim 1, wherein, Constructing a fixed-depth prefix tree includes: A variable token filtering mechanism is set for the prefix tree so that, during the matching phase, the variables contained in the multi-source logs are matched with a second preset wildcard. Set an upper limit on the number of child nodes for each node in the prefix tree.

5. The method according to any one of claims 1 to 4, characterized in that, Based on the log sequences in the preprocessed multi-source logs, clustering is performed on the preprocessed multi-source logs using the fixed-depth prefix tree and the longest common subsequence, including: Determine the LCS of each log sequence in the preprocessed multi-source logs with each log template sequence in the log group under each leaf node in the prefix tree; The length of the LCS is calculated as the ratio of the length of the longer sequence among the corresponding log sequence in each log sequence of the multi-source logs and the length of each log template sequence, and is used as the similarity score. ； in, , representing the sequence of logs to be matched in a multi-source log; , indicating a log template sequence, , Sequences Length; This represents the maximum length of two sequences; Indicates the length of the LCS; Based on the ratio, target log groups with the highest similarity value to each log sequence are selected; The highest similarity value is compared with a predefined threshold. If the highest similarity value is greater than the predefined threshold, the target log group is returned as the best matching result; if the highest similarity value is less than or equal to the predefined threshold, it indicates that no suitable log group was found.

6. The method according to claim 5, characterized in that, If no log group exists under any leaf node in the prefix tree, or if no matching log group is found, a new log group identifier is generated according to preset rules, a new log group is created using the current log as the initial template, and the prefix tree is updated. The process includes generating a new log group identifier based on preset rules, creating a new log group with the current log as the initial template, and updating the prefix tree. This includes taking the preset first N tokens obtained after processing the current log with preset word segmentation rules as internal nodes of the prefix tree, and storing the template of the current log and the current log identifier ID as associated data in the leaf nodes of the path, where N is a positive integer.

7. The method according to claim 5, characterized in that, The method further includes: after returning the target log group as the best matching result when the highest similarity value is greater than the predefined threshold, updating the prefix tree. The update prefix tree includes: adding the current log ID to the associated ID list of the corresponding log group, and updating the log template. The update log template includes: comparing the token sequence of the current log with the token sequence of the original template of the log group, solving the LCS of the token sequence of the current log with the token sequence of the original template of the log group through a dynamic programming algorithm, and using the LCS as a benchmark, replacing the part of the longer sequence in the token sequence of the current log with the token sequence of the original template of the log group that does not match the LCS with a preset wildcard.

8. A log clustering device based on the longest common subsequence, characterized in that, The device includes: The log reading module is used to read logs from multiple sources. The log preprocessing module is used to: preprocess the multi-source logs; The log clustering module is used to: construct a fixed-depth prefix tree, and cluster the preprocessed multi-source logs based on each log sequence in the preprocessed multi-source logs using the fixed-depth prefix tree and the longest common subsequence (LCS). The storage module is used to: determine the storage dimension of the clustering results of the preprocessed multi-source logs, and store the multi-source logs and the clustering results.

9. A log clustering device based on the longest common subsequence, characterized in that, The device includes: At least one processor; At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions implementing the method of any one of claims 1 to 7 when executed by the at least one processor.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 7.

Images