System log deduplication and correlation analysis method and device based on semantic similarity
By using a log deduplication and correlation analysis method based on semantic similarity, the problems of insufficient log deduplication accuracy and inaccurate correlation judgment are solved, achieving efficient and accurate log processing and interpretability of results, which is suitable for large-scale log data analysis.
Patent Information
- Application Number
- CN202610844193.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-11
- Publication Date
- 2026-08-25
AI Technical Summary
Existing technologies for log deduplication and correlation analysis suffer from insufficient deduplication accuracy, inaccurate correlation judgment, low processing efficiency, and lack of interpretability, especially when dealing with massive log data, making it difficult to meet the requirements of real-time performance and accuracy.
A log deduplication and correlation analysis method based on semantic similarity is adopted. By obtaining the log file path, extracting and preprocessing the log data, generating a structured log list, calculating the semantic similarity between logs, and combining a greedy grouping algorithm and a dual-mode analysis module, efficient deduplication and correlation analysis are achieved.
It significantly improves log deduplication accuracy and compression rate, enhances the accuracy of correlation judgment and processing efficiency, and strengthens the interpretability of results, enabling the processing of large amounts of log data in a short time.
Smart Images
Figure CN122633642A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of log data processing technology, specifically to a method and apparatus for deduplication and correlation analysis of system logs based on semantic similarity. Background Technology
[0002] In the operation and maintenance of large-scale IT infrastructure such as storage systems and cloud computing platforms, system logs are the core data source for fault diagnosis and root cause analysis. As system scale expands, log data grows explosively, with a single fault event potentially generating tens or even hundreds of thousands of log records. These massive amounts of logs contain a large amount of duplicate and redundant information, posing a significant challenge to operations and maintenance personnel in fault localization and analysis.
[0003] Currently, existing technologies mainly address this issue by using methods such as log deduplication based on string matching, relevance filtering based on keywords, and log classification based on traditional clustering to remove a large amount of duplicate and redundant information in massive logs.
[0004] Log deduplication based on string matching: This method uses exact string matching or simple edit distance algorithm to deduplicat logs. Its core lies in literal similarity judgment, which measures the similarity between two logs by indicators such as Levenshtein distance.
[0005] Keyword-based relevance filtering: This method is designed for fault diagnosis scenarios and uses keyword matching to filter relevant logs. It relies on a manually maintained rule base and uses Boolean matching or simple word frequency statistics to determine relevance.
[0006] Log classification based on traditional clustering: This method uses algorithms such as K-means and hierarchical clustering to classify logs. It requires pre-setting the number of clusters K and is sensitive to the initial centroid, making it easy to get trapped in local optima.
[0007] However, existing technologies have the following drawbacks:
[0008] 1. Insufficient deduplication accuracy
[0009] String matching-based methods can only identify logs that are literally identical or highly similar, and cannot handle cases where the semantics are similar but the wording is different. For example, "Controller A panic detected" and "Detected that controller A panic occurred" have the same semantics but are significantly different in wording. Traditional methods cannot identify them as duplicates, resulting in a large amount of semantically redundant logs remaining after deduplication, and a low compression rate (usually <30%).
[0010] 2. Inaccurate correlation assessment
[0011] Keyword-based methods cannot understand the contextual semantics of logs and are prone to misjudgments. For example, "ERROR: Connection timeout" and "ERROR: Memory overflow" both contain the keyword ERROR, but their relevance to specific faults may be completely different. Relying on manually maintained rule bases makes it difficult to cover all fault modes and results in poor adaptability.
[0012] 3. Low processing efficiency
[0013] Traditional clustering algorithms require multiple iterations to calculate the distance matrix, resulting in high time complexity (O(n²) or higher). When dealing with tens of thousands of logs, the processing time can reach several minutes or even tens of minutes, which cannot meet real-time requirements.
[0014] 4. Lack of explainability
[0015] Traditional methods provide correlation scores or clustering results that lack detailed explanations, making it difficult for operations and maintenance personnel to understand why a particular log entry is considered highly relevant or belongs to a specific cluster, which is detrimental to fault analysis and decision-making. Summary of the Invention
[0016] To address these issues, this application provides a method and apparatus for deduplication and correlation analysis of system logs based on semantic similarity, thereby solving the problems of insufficient deduplication accuracy, inaccurate correlation judgment, low processing efficiency, and lack of interpretability in existing log deduplication methods.
[0017] To achieve the above objectives, this application provides the following technical solution:
[0018] Firstly, a method for deduplication and correlation analysis of system logs based on semantic similarity includes:
[0019] Step 1: Obtain the list of original log file paths, extract the original log data according to the list of original log file paths, and perform data preprocessing on the original log data to obtain a structured log list;
[0020] Step 2: Extract the semantic features of each log text in the structured log list and generate the corresponding log feature representation; the log feature representation is used to measure the semantic similarity between logs;
[0021] Step 3: Based on the log feature representation, calculate the semantic similarity between logs according to the preset similarity measurement algorithm, and deduplicate the logs according to the semantic similarity to output a deduplicated log list; wherein, the deduplicated log list contains at least representative logs and their corresponding repetition frequency information;
[0022] Step 4: Obtain the fault description text input by the user; if the fault description text is not empty, take the deduplicated log list and the repetition frequency information as input, and execute the correlation analysis mode to output the log list sorted by correlation; if the fault description text is empty, take the deduplicated log list and the repetition frequency information as input, and execute the clustering analysis mode to output the clustering result list.
[0023] Step 5: Generate a structured analysis report based on the log list sorted by relevance or the clustering result list.
[0024] Preferably, in step 1, when extracting the original log data based on the original log file path list, a full extraction mode or an ERROR extraction mode is used, wherein the ERROR extraction mode uses regular expressions or a log parsing library.
[0025] Preferably, in step 3, the similarity measurement algorithm adopts the Jaccard similarity calculation method based on term set, the Cosine similarity algorithm, the edit distance similarity algorithm, the longest common subsequence similarity algorithm, the cosine distance between vectors after combining TF-IDF vectorization, or the semantic similarity is calculated after generating sentence embeddings using a pre-trained Sentence-BERT model.
[0026] Preferably, when the similarity measurement algorithm adopts the Jaccard similarity calculation method based on term sets, it specifically includes: converting each log text to lowercase and segmenting it by spaces to generate a corresponding term set; for any two logs, calculating the ratio of the intersection size to the union size of the term sets as the Jaccard similarity between the two logs; and determining that two logs with a Jaccard similarity exceeding a preset threshold are semantically similar and are classified into the same similarity group.
[0027] Preferably, the preset threshold is 0.5 by default.
[0028] Preferably, in step 3, when deduplicating logs based on the semantic similarity, a greedy grouping algorithm, a density-based clustering algorithm, a minimum hash algorithm combined with a locality-sensitive hashing algorithm, or hierarchical clustering is used to construct a log similarity tree for deduplication.
[0029] Preferably, when using the greedy grouping algorithm for deduplication, each log entry is compared with the representative log entries of each existing group in turn for similarity. When the similarity first reaches or exceeds a preset threshold, the log entry is assigned to the corresponding group and the comparison is stopped.
[0030] Preferably, in step 4, the relevance analysis mode includes: inputting the deduplicated log list and the repetition frequency information into the TF-IDF model combined with a pre-trained classifier or into a large language model in batches, and obtaining a log list sorted by relevance through relevance probability prediction or semantic understanding scoring, respectively.
[0031] Preferably, in step 4, the clustering analysis mode includes: inputting the deduplicated log list and the repetition frequency information in batches into the TF-IDF combined with DBSCAN clustering algorithm or into a large language model, and obtaining a clustering result list by means of vectorized clustering or semantic pattern recognition grouping, respectively.
[0032] Secondly, a system log deduplication and correlation analysis device based on semantic similarity includes:
[0033] The log preprocessing module is used to obtain a list of raw log file paths, extract raw log data based on the list of raw log file paths, and perform data preprocessing on the raw log data to obtain a structured log list.
[0034] The semantic deduplication module is used to extract the semantic features of each log text in the structured log list and generate a corresponding log feature representation; the log feature representation is used to measure the semantic similarity between logs.
[0035] Based on the log feature representation, the semantic similarity between logs is calculated according to a preset similarity measurement algorithm, and the logs are deduplicated according to the semantic similarity to output a deduplicated log list; wherein the deduplicated log list contains at least representative logs and their corresponding repetition frequency information;
[0036] The dual-mode analysis module is used to obtain the fault description text input by the user; if the fault description text is not empty, the deduplicated log list and the repetition frequency information are used as input, and the correlation analysis mode is executed to output the log list sorted by correlation; if the fault description text is empty, the deduplicated log list and the repetition frequency information are used as input, and the clustering analysis mode is executed to output the clustering result list.
[0037] And generate a structured analysis report based on the log list sorted by relevance or the clustering result list.
[0038] Compared with the prior art, this application has at least the following beneficial effects:
[0039] 1. This application provides a method for deduplication and correlation analysis of system logs based on semantic similarity, including: obtaining a list of original log file paths and extracting the original log count to obtain a structured log list; extracting the semantic features of each log text in the structured log list and generating a corresponding log feature representation; calculating the semantic similarity between logs based on the log feature representation according to a preset similarity measurement algorithm, and deduplicating the logs according to the semantic similarity, outputting a deduplicated log list; obtaining fault description text input by the user, performing correlation analysis mode or clustering analysis mode according to the fault description text, and obtaining a log list or clustering result list sorted by correlation; generating a structured analysis report based on the log list or clustering result list sorted by correlation. This application significantly improves log deduplication accuracy, correlation judgment accuracy, processing efficiency, and result interpretability through an architecture combining semantic deduplication and dual-mode analysis.
[0040] 2. The term-level Jaccard similarity algorithm is adopted, which significantly improves the deduplication and compression rate.
[0041] 3. The adoption of a greedy grouping algorithm further significantly improves the processing efficiency of large-scale logs. Attached Figure Description
[0042] To more intuitively illustrate the prior art and this application, exemplary drawings are provided below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary drawings, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).
[0043] Figure 1 A flowchart of a system log deduplication and correlation analysis method based on semantic similarity provided in Embodiment 1 of this application;
[0044] Figure 2 This is a schematic diagram of the structure of a system log deduplication and correlation analysis method based on semantic similarity provided in Embodiment 1 of this application;
[0045] Figure 3 The flowchart for semantic-level deduplication provided in Embodiment 1 of this application. Detailed Implementation
[0046] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0047] In the description of this application: unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," "third," etc., in this application are intended to distinguish the objects referred to and do not have any special meaning in terms of technical connotation (e.g., they should not be construed as an emphasis on importance or order). Expressions such as "including," "comprising," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).
[0048] The terms used in this application, such as "upper," "lower," "left," "right," and "middle," are generally used to indicate the general relative positional relationship for the purpose of intuitive understanding by referring to the accompanying drawings, and are not absolute limitations on the positional relationship in the actual product.
[0049] Example 1
[0050] Please see Figure 1 and Figure 2 This embodiment provides a method for deduplication and relevance analysis of system logs based on semantic similarity. This method employs a hybrid approach of "term-level similarity deduplication + dual-mode intelligent analysis," including:
[0051] S1: Obtain the list of original log file paths, extract the original log data based on the list of original log file paths, and perform data preprocessing on the original log data to obtain a structured log list;
[0052] Specifically, this step, when extracting raw log data based on the list of raw log file paths, supports two methods: full extraction mode and ERROR (i.e., level-filtered) extraction mode. In full extraction mode, all log lines are read; in ERROR extraction mode, regular expressions or log parsing libraries (e.g., Drain, Spell) are used. When using regular expressions, r'\b(ERROR)\b' only extracts logs containing the ERROR level.
[0053] This step involves filtering data based on multiple dimensions such as log level (INFO, WARN, ERROR, FATAL), time range, and source host during data preprocessing. The output structured log list can be represented as: [(log_text, line_number, file_path), ...].
[0054] It should be noted that this embodiment supports real-time log stream (such as Kafka, Flume) input, rather than only file input.
[0055] S2: Extract the semantic features of each log text in the structured log list and generate the corresponding log feature representation; the log feature representation is used to measure the semantic similarity between logs;
[0056] S3: Based on log feature representation, calculate the semantic similarity between logs according to the preset similarity measurement algorithm, and deduplicate the logs according to the semantic similarity, and output the deduplicated log list; wherein, the deduplicated log list contains at least representative logs and their corresponding repetition frequency information;
[0057] Specifically, in this step, the similarity measurement algorithm adopts the Jaccard similarity calculation method based on term set, the Cosine similarity algorithm, the Levenshtein Distance similarity algorithm, the Longest Common Subsequence (LCS) similarity algorithm, the cosine distance between vectors after combining TF-IDF vectorization, or the semantic similarity is calculated after generating sentence embeddings using a pre-trained Sentence-BERT model.
[0058] For more details, please see Figure 3 When the similarity measurement algorithm uses the Jaccard similarity calculation method based on term sets, it includes:
[0059] S301: Convert each log text to lowercase and segment it by spaces to generate a corresponding term set (e.g., {"error", "controller", "panic", "detected"}).
[0060] S302: For any two log entries, calculate the ratio of the intersection size to the union size of the term sets, which is taken as the Jaccard similarity between the two log entries. The Jaccard coefficient formula is: similarity = |A ∩ B| / |A ∪ B| (that is, the intersection size divided by the union size, which can also be expressed as: similarity = |intersection| / |union|).
[0061] S303: Two log entries with a Jaccard similarity exceeding a preset threshold are considered semantically similar and grouped into the same similarity group, thereby automatically removing duplicate terms within a single log entry and retaining unique terms. The preset threshold is 0.5 by default. A threshold of 0.5 means that two log entries are considered similar if at least 50% of their terms overlap. For example, the intersection of the terms "Controller A panicdetected" and "Controller A panicked" is {"controller", "a", "panic"}, and the union is {"controller", "a", "panic", "detected", "controller", "occurred"}, with a similarity of 3 / 6 = 0.5.
[0062] In this step, when deduplicating the logs according to semantic similarity, a greedy grouping algorithm, a density-based clustering algorithm, a minhash algorithm combined with a locality-sensitive hashing algorithm (MinHash + LSH), or hierarchical clustering (HierarchicalClustering) is used to construct a log similarity tree for deduplication.
[0063] More specifically, when using the greedy grouping algorithm for deduplication, the similarity of each log is compared with the representative logs of each existing group in turn. When the similarity first reaches or exceeds the preset threshold, the log is classified into the corresponding group and the comparison stops.
[0064] The greedy grouping algorithm includes:
[0065] Incremental matching: Each new log is compared with the representative rows of the existing groups one by one, and the first group with a similarity ≥ threshold is joined.
[0066] Early stopping mechanism: Stop searching immediately after successful matching to avoid traversing all groups.
[0067] Time optimization: The complexity is reduced from the theoretical O(n²) to the actual O(n×k), where k is the number of groups (k << n), significantly improving the processing efficiency.
[0068] In this step, the deduplicated log list can be represented as: [{"log": "log content", "count": number of repetitions},...].
[0069] S4: Obtain the fault description text input by the user; if the fault description text is not empty, use the deduplicated log list and the repetition frequency information as inputs, and execute the correlation analysis mode to output a log list sorted by correlation; if the fault description text is empty, use the deduplicated log list and the repetition frequency information as inputs, and execute the clustering analysis mode to output a clustering result list.
[0070] Specifically, in this step, the discriminant logic of the dual-mode adaptive decision is as follows:
[0071] Input detection: Determine whether the query parameter is None or an empty string. <The relevance analysis mode includes: inputting the deduplicated log list and repetition frequency information into the TF-IDF model in batches and combining it with a pre-trained classifier or inputting it into a large language model, and obtaining a log list sorted by relevance through relevance probability prediction or semantic understanding scoring, respectively.
[0075] In other words, the correlation analysis model (with fault description query):
[0076] Triggering condition: The user provides a description of the fault.
[0077] Analysis path:
[0078] Path 1 (TF-IDF model): Use a pre-trained classifier to predict the probability of each log being associated with a fault;
[0079] Path 2 (LLM model): Call the large language model for semantic understanding and relevance scoring; the structured Prompt project for relevance scoring is as follows:
[0080] [Fault Description]: {query};
[0081] [Log List]: 0. ERROR Controller panic (occurrence count: 3) 1. ERROR Disktimeout (occurrence count: 5)...;
[0082] [Scoring Criteria]: 0.9-1.0: Highly relevant (weight: 50% direct description of the fault phenomenon + 50% containing root cause keywords); 0.7-0.9: Moderately relevant (weight: 40% relevant information + 60% time-series relevance); 0.5-0.7: Lowly relevant (weight: 100% indirect relationship);
[0083] [Output format]: JSON array [{"index":0, "relevance_score":0.95, "reason":"..."}].
[0084] Response parsing process:
[0085] Two-layer extraction: first extract <answer>Tags, then extract the "json" code block;
[0086] Index mapping: Map the score back to the original log based on the returned index field;
[0087] Fault tolerance and degradation: When JSON parsing fails, assign a default score of 0.5 to all logs in that batch and record the reason "Response parsing failed".
[0088] LLM unsupervised clustering mechanisms include:
[0089] Semantic pattern recognition;
[0090] Multi-dimensional feature extraction:
[0091] Error type features (such as panic, timeout, overflow) account for 40% of the weight;
[0092] Component features (such as controller, disk, network) account for 30% of the weight;
[0093] Operational features (such as read, write, restart) account for 30% of the weight;
[0094] Adaptive clustering number: No need to preset the K value, it is automatically determined by the LLM based on the log complexity (usually 3-10 clusters);
[0095] Result reconstruction;
[0096] Log aggregation: Extract logs from the original list based on log_indices and calculate the total count;
[0097] Priority sorting: Clusters are sorted in descending order of count, with the error categories that have the greatest impact displayed first;
[0098] Enhanced interpretability: Each cluster includes a description and key_features, making it easier for operations and maintenance personnel to quickly understand.
[0099] Output: A list of logs sorted in descending order of relevance.
[0100] In this step, the correlation analysis mode performs secondary routing based on the llm_mode parameter when selecting paths:
[0101] llm_mode = True: LLM deep semantic analysis (high accuracy, long processing time);
[0102] llm_mode = False: Fast TF-IDF vector prediction (high efficiency, low cost).
[0103] Clustering analysis modes include: inputting the deduplicated log list and repetition frequency information in batches into the TF-IDF combined with DBSCAN clustering algorithm or into a large language model, and obtaining a list of clustering results through vectorized clustering or semantic pattern recognition grouping, respectively.
[0104] In other words, clustering analysis pattern (without fault description):
[0105] Triggering condition: The user did not provide a description of the fault;
[0106] Analysis path:
[0107] Path 1 (TF-IDF+DBSCAN): Uses a vectorized clustering algorithm;
[0108] Path 2 (LLM clustering): Invoke the large language model to identify and group error patterns in the logs;
[0109] Output: A list of clustering results.
[0110] It should be noted that, in this embodiment, the dynamic weight allocation is as follows:
[0111] Correlation pattern: TF-IDF vectorization 40% + classifier probability prediction 60%;
[0112] Clustering pattern: semantic similarity 70% + key feature extraction 30%.
[0113] S5: Generate a structured analysis report based on a log list or clustering results list sorted by relevance.
[0114] The system log deduplication and correlation analysis method based on semantic similarity provided in this embodiment has the following beneficial effects:
[0115] 1. Significantly improve deduplication accuracy and compression rate: Employs term-level Jaccard similarity algorithm, which can identify logs that are semantically similar but have different expressions (such as mixed Chinese and English, synonym replacement, etc.). Compared with traditional string matching methods, the deduplication and compression rate is increased from 30% to over 70%. In actual testing, after deduplication of 6840 fault logs, 450 representative logs were retained, effectively reducing the analysis burden on operation and maintenance personnel.
[0116] 2. Improve the accuracy of relevance judgment: By understanding the semantic content and context of logs through LLM, the reliance on manual rule base is eliminated. Compared with keyword-based methods, the accuracy of relevance judgment is improved by more than 40%. Detailed reasons for the relevance of each log are provided, which enhances the interpretability of the results and makes it easier for operations and maintenance personnel to quickly locate key logs.
[0117] 3. Achieve a unified dual-mode analysis framework: By automatically determining whether there is a fault description (query), it intelligently switches between correlation analysis and cluster analysis modes. A single system supports both supervised and unsupervised scenarios, reducing the cost of switching tools. It provides two implementation paths, TF-IDF and LLM, allowing for flexible selection based on actual needs (offline efficiency vs. online high accuracy).
[0118] 4. Improve the efficiency of large-scale log processing: Adopt a batch processing strategy (batch_size=1000) to break down large-scale logs into multiple small batches. When processing tens of thousands of logs, memory usage is reduced by 70% compared to full processing. Through progress prompts and early stop optimization, fast response (usually <2 minutes) is achieved while ensuring accuracy.
[0119] 5. Enhance the interpretability and operability of analysis results: The LLM mode provides a relevance_reason for each log entry, explaining why it is determined to be highly / lowly relevant; the clustering mode generates descriptions and key_features to help operations and maintenance personnel quickly understand the error category; statistical information (total_logs, deputed_logs, relevant_logs, etc.) provides a complete data traceability chain.
[0120] In summary, this embodiment significantly improves log deduplication accuracy, relevance judgment accuracy, processing efficiency, and result interpretability through an architecture that combines semantic deduplication with dual-mode analysis.
[0121] Example 2
[0122] This embodiment provides a system log deduplication and relevance analysis device based on semantic similarity. The device adopts a hybrid architecture of "term-level similarity deduplication + dual-mode intelligent analysis", and specifically includes the following core modules:
[0123] The log preprocessing module is used to obtain a list of raw log file paths, extract raw log data based on the list of raw log file paths, and perform data preprocessing on the raw log data to obtain a structured log list.
[0124] The semantic deduplication module is used to extract the semantic features of each log text in the structured log list and generate the corresponding log feature representation; the log feature representation is used to measure the semantic similarity between logs.
[0125] Based on log feature representation, semantic similarity between logs is calculated according to a preset similarity measurement algorithm, and logs are deduplicated according to semantic similarity, and a deduplicated log list is output; wherein, the deduplicated log list contains at least representative logs and their corresponding repetition frequency information;
[0126] The dual-mode analysis module is used to obtain the fault description text input by the user; if the fault description text is not empty, the deduplicated log list and repetition frequency information are used as input, and the correlation analysis mode is executed to output the log list sorted by correlation; if the fault description text is empty, the deduplicated log list and repetition frequency information are used as input, and the clustering analysis mode is executed to output the clustering result list.
[0127] It also generates structured analysis reports based on log lists or clustering results lists sorted by relevance.
[0128] For details on the implementation of each module in a semantic similarity-based system log deduplication and correlation analysis device, please refer to the above description of the limitations of a semantic similarity-based system log deduplication and correlation analysis method, which will not be repeated here.
[0129] The technical features of the above embodiments can be combined in any way (as long as there is no contradiction in the combination of these technical features). For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; these embodiments not explicitly written should also be considered to be within the scope of this specification.< / answer>
Claims
1. A method for deduplication and correlation analysis of system logs based on semantic similarity, characterized in that, include: Step 1: Obtain the list of original log file paths, extract the original log data according to the list of original log file paths, and perform data preprocessing on the original log data to obtain a structured log list; Step 2: Extract the semantic features of each log text in the structured log list and generate the corresponding log feature representation; the log feature representation is used to measure the semantic similarity between logs; Step 3: Based on the log feature representation, calculate the semantic similarity between logs according to the preset similarity measurement algorithm, and deduplicate the logs according to the semantic similarity to output a deduplicated log list; wherein, the deduplicated log list contains at least representative logs and their corresponding repetition frequency information; Step 4: Obtain the fault description text input by the user; if the fault description text is not empty, take the deduplicated log list and the repetition frequency information as input, and execute the correlation analysis mode to output the log list sorted by correlation; if the fault description text is empty, take the deduplicated log list and the repetition frequency information as input, and execute the clustering analysis mode to output the clustering result list. Step 5: Generate a structured analysis report based on the log list sorted by relevance or the clustering result list.
2. The method for deduplication and correlation analysis of system logs based on semantic similarity according to claim 1, characterized in that, In step 1, when extracting raw log data based on the list of raw log file paths, either a full extraction mode or an ERROR extraction mode is used. The ERROR extraction mode uses regular expressions or a log parsing library.
3. The method for deduplication and correlation analysis of system logs based on semantic similarity according to claim 1, characterized in that, In step 3, the similarity measurement algorithm adopts the Jaccard similarity calculation method based on term set, the Cosine similarity algorithm, the edit distance similarity algorithm, the longest common subsequence similarity algorithm, the cosine distance between vectors after combining TF-IDF vectorization, or the semantic similarity is calculated after generating sentence embeddings using a pre-trained Sentence-BERT model.
4. The method for deduplication and correlation analysis of system logs based on semantic similarity according to claim 3, characterized in that, When the similarity measurement algorithm adopts the Jaccard similarity calculation method based on term sets, it specifically includes: converting each log text to lowercase and segmenting it by spaces to generate a corresponding term set; for any two logs, calculating the ratio of the intersection size to the union size of the term sets as the Jaccard similarity between the two logs; and determining that two logs with Jaccard similarity exceeding a preset threshold are semantically similar and are classified into the same similarity group.
5. The method for deduplication and correlation analysis of system logs based on semantic similarity according to claim 4, characterized in that, The preset threshold is 0.5 by default.
6. The method for deduplication and correlation analysis of system logs based on semantic similarity according to claim 1, characterized in that, In step 3, when deduplicating logs based on the semantic similarity, a greedy grouping algorithm, a density-based clustering algorithm, a minimum hash algorithm combined with a locality-sensitive hashing algorithm, or hierarchical clustering are used to construct a log similarity tree for deduplication.
7. The method for deduplication and correlation analysis of system logs based on semantic similarity according to claim 6, characterized in that, When using a greedy grouping algorithm for deduplication, each log entry is compared with the representative log entries of each existing group in turn to determine their similarity. When the similarity first reaches or exceeds a preset threshold, the log entry is assigned to the corresponding group and the comparison is stopped.
8. The method for deduplication and correlation analysis of system logs based on semantic similarity according to claim 1, characterized in that, In step 4, the relevance analysis mode includes: inputting the deduplicated log list and the repetition frequency information into the TF-IDF model combined with the pre-trained classifier or into the large language model in batches, and obtaining the log list sorted by relevance by means of relevance probability prediction or semantic understanding scoring, respectively.
9. The method for deduplication and correlation analysis of system logs based on semantic similarity according to claim 1, characterized in that, In step 4, the clustering analysis mode includes: inputting the deduplicated log list and the repetition frequency information in batches into the TF-IDF combined with DBSCAN clustering algorithm or into a large language model, and obtaining a clustering result list by means of vectorized clustering or semantic pattern recognition grouping, respectively.
10. A system log deduplication and correlation analysis device based on semantic similarity, characterized in that, include: The log preprocessing module is used to obtain a list of raw log file paths, extract raw log data based on the list of raw log file paths, and perform data preprocessing on the raw log data to obtain a structured log list. The semantic deduplication module is used to extract the semantic features of each log text in the structured log list and generate a corresponding log feature representation; the log feature representation is used to measure the semantic similarity between logs. Based on the log feature representation, the semantic similarity between logs is calculated according to a preset similarity measurement algorithm, and the logs are deduplicated according to the semantic similarity to output a deduplicated log list; wherein the deduplicated log list contains at least representative logs and their corresponding repetition frequency information; The dual-mode analysis module is used to obtain the fault description text input by the user; if the fault description text is not empty, the deduplicated log list and the repetition frequency information are used as input, and the correlation analysis mode is executed to output the log list sorted by correlation; if the fault description text is empty, the deduplicated log list and the repetition frequency information are used as input, and the clustering analysis mode is executed to output the clustering result list. And generate a structured analysis report based on the log list sorted by relevance or the clustering result list.