Log anomaly diagnosis and repair method and system based on knowledge base and large model collaboration
By employing a log anomaly diagnosis and repair method that integrates a knowledge base and a large model, the problems of large data volume, difficulty in anomaly identification, and reliance on experience in log analysis are solved. This approach enables efficient and accurate log anomaly diagnosis and repair, improving the system's automation and adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUJIAN XINGWANG INTELLIGENT SOFTWARE CO LTD
- Filing Date
- 2025-12-09
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies for log analysis suffer from problems such as large and scattered data volume, difficulty in anomaly identification, reliance on experience for repair, and lack of context, resulting in poor analysis accuracy, low automation, and insufficient generalization ability.
A log anomaly diagnosis and repair method based on knowledge base and large model collaboration is adopted. Through multi-dimensional log pre-analysis, vector database and large model inference framework, log automatic processing, anomaly judgment and repair suggestion generation are realized.
It improves the automation, accuracy, and adaptability of log analysis, enhances the accuracy of anomaly identification, generates actionable remediation suggestions, improves processing efficiency, and strengthens interpretability.
Smart Images
Figure CN122019219A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of log analysis technology, and in particular to a method and system for diagnosing and repairing log anomalies based on the collaboration of a knowledge base and a large model. Background Technology
[0002] In modern complex software systems, logs are crucial for diagnosing system anomalies and pinpointing the root causes of failures. However, log data analysis typically presents the following challenges: Large and dispersed data volume: Systems often deploy multiple log collectors, generating massive amounts of heterogeneous logs; Anomaly identification is difficult: not all logs are "real anomalies," and it is necessary to distinguish between warnings, debugging messages, and actual faults; Repair relies on experience: Even if an anomaly is identified, the repair solution is highly dependent on the experience of the operations and maintenance personnel and lacks standardized and automated methods; Missing context: A single log line is insufficient to reflect the full picture of the fault; a comprehensive judgment must be made by combining the historical error context.
[0003] In existing technologies, some solutions use rule engines or simple keyword matching for anomaly detection, but their generalization ability is poor; others have attempted to introduce machine learning models for classification, but they lack a deep understanding of log semantics and the ability to generate repair suggestions. In addition, existing knowledge bases are mostly static rule bases, making it difficult to dynamically adapt to new anomaly patterns.
[0004] Therefore, there is a need for an intelligent log analysis method and system that can automatically determine whether a log is a real anomaly and generate actionable repair suggestions by combining historical knowledge and large model reasoning capabilities. Summary of the Invention
[0005] The technical problem to be solved by this invention is to provide a log anomaly diagnosis and repair method and system based on the collaboration of knowledge base and large model. By introducing a multi-dimensional log pre-analysis mechanism, a knowledge base structure that supports pre-anomaly association, and a collaborative reasoning framework of large model and vector retrieval, the invention effectively overcomes the shortcomings of existing technologies in terms of accuracy, generalization, automation and intelligence.
[0006] Firstly, this invention provides a log anomaly diagnosis and repair method based on the collaboration of a knowledge base and a large model, including: Log collection and processing process: Obtain system operation logs from different sources, process and summarize them in a unified format; Log pre-analysis process: Logs are segmented, dimensionally split, and preliminarily analyzed; Log knowledge base matching process: Based on the preliminary analysis results, multi-dimensional matching is performed between the vector database and the log knowledge base; Large model inference process: Based on the matching results and the current log content, generate anomaly judgment and repair suggestions.
[0007] Furthermore, the vector database includes vector representations of exception log examples, vector representations of exception log descriptions, and corresponding exception log IDs; the structure of the log knowledge base includes exception log IDs, exception log examples, exception repair steps, exception log descriptions, preceding exception log IDs, and preceding exception log descriptions.
[0008] Furthermore, the log pre-analysis process specifically includes: splitting the log file into multiple smaller files, performing semantic parsing on each line of the log using a large language model, and outputting the following structured dimensions: Source log: The original log text; Log pre-attribution problem: Large models summarize the possible fault categories to which a log might belong based on their own knowledge; Log preprocessing method: Preliminary suggested repair directions for large models; Previous error source log: Records the log content of the previous line that was judged to be an error, used for context association; Timestamp: Extracts time information from the log; if it is missing, it is marked as missing.
[0009] Furthermore, the log knowledge base matching process specifically includes: The "log pre-attribution problem" is vectorized, and similar anomaly descriptions are retrieved from the vector database to return a list of IDs and the first similarity score. The "log pre-method" is vectorized, and similar repair descriptions are retrieved from the vector database to return a list of IDs and a second similarity score. Vectorize the "previous error source log", retrieve the preceding anomaly from the vector database, and return a list of IDs and a third similarity score; By combining the three similarity scores, a comprehensive matching ranking is generated; Based on the Top-N matching results, retrieve the corresponding repair steps from the log knowledge base.
[0010] Furthermore, the large model reasoning process specifically includes: inputting the current log, the matched repair steps, and context information into the large model to generate the final diagnostic conclusion and executable repair suggestions.
[0011] Secondly, this invention provides a log anomaly diagnosis and repair system based on the collaboration of a knowledge base and a large model, comprising: The log collection and processing module is used to acquire system operation logs from different sources, process and summarize them in a unified format; The log pre-analysis module is used to segment, dimensionally split, and perform preliminary analysis on logs. The log knowledge base matching module is used to perform multi-dimensional matching between the vector database and the log knowledge base based on the preliminary analysis results. The large model inference module is used to generate anomaly detection and repair suggestions based on the matching results and the current log content.
[0012] Furthermore, the vector database includes vector representations of exception log examples, vector representations of exception log descriptions, and corresponding exception log IDs; the structure of the log knowledge base includes exception log IDs, exception log examples, exception repair steps, exception log descriptions, preceding exception log IDs, and preceding exception log descriptions.
[0013] Furthermore, the log pre-analysis module specifically includes: The splitting submodule is used to split log files into multiple smaller files; The structured submodule is used to perform semantic parsing on each line of log data using the large language model, outputting the following structured dimensions: Source log: The original log text; Log pre-attribution problem: Large models summarize the possible fault categories to which a log might belong based on their own knowledge; Log preprocessing method: Preliminary suggested repair directions for large models; Previous error source log: Records the log content of the previous line that was judged to be an error, used for context association; Timestamp: Extracts time information from the log; if it is missing, it is marked as missing.
[0014] Furthermore, the log knowledge base matching module specifically includes: The first vectorization submodule is used to vectorize the "log pre-attribution problem", retrieve similar anomaly descriptions from the vector database, and return a list of IDs and the first similarity score; The second vectorization submodule is used to vectorize the "log pre-method", retrieve similar repair descriptions from the vector database, and return a list of IDs and a second similarity score. The third vectorization submodule is used to vectorize the "previous error source log", retrieve the previous anomaly from the vector database, and return a list of IDs and a third similarity score. The comprehensive ranking submodule is used to fuse the three similarity scores to generate a comprehensive matching ranking; The repair step retrieval submodule is used to retrieve the corresponding repair steps from the log knowledge base based on the Top-N matching results.
[0015] Furthermore, the large model inference module is specifically used to input the current log, the matched repair steps, and context information into the large model to generate the final diagnostic conclusion and executable repair suggestions.
[0016] The technical solutions provided in the embodiments of the present invention have at least the following technical effects: 1. High degree of automation: The entire process, from log collection to generating remediation suggestions, is automated; 2. High analytical accuracy: Combining multi-dimensional matching and large-scale model reasoning improves the accuracy of anomaly identification; 3. High adaptability: The knowledge base can be dynamically updated to adapt to new and unusual scenarios; 4. High efficiency: The pre-analysis module segments logs, improving the processing efficiency of large models; 5. Good interpretability: Each analysis step has structured output, facilitating traceability and optimization.
[0017] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description
[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0019] Figure 1 This is a flowchart illustrating the overall process of the method in Embodiment 1 of the present invention. Figure 2 This is a flowchart of the log pre-analysis process in Embodiment 1 of the present invention; Figure 3 This is a flowchart of the log knowledge base matching process in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the operation process in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the system structure in Embodiment 2 of the present invention; Figure 6 This is a schematic diagram of the log knowledge base matching module in Embodiment 2 of the present invention. Detailed Implementation
[0020] This application provides a log anomaly diagnosis and repair method and system based on knowledge base and large model collaboration. By introducing a multi-dimensional log pre-analysis mechanism, a knowledge base structure that supports pre-anomaly association, and a large model and vector retrieval collaborative reasoning framework, it effectively overcomes the shortcomings of existing technologies in terms of accuracy, generalization, automation, and intelligence.
[0021] The overall concept of the technical solution in this application is as follows: According to the inventor's analysis, the existing technology has the above-mentioned problems mainly due to the following reasons: Anomaly detection relies on static rules and has poor generalization ability. Most mainstream log analysis systems currently use keyword matching, regular expressions, or predefined rules (such as alert rules in ELKStack) to identify anomalies. These methods cannot understand log semantics, have low recognition rates for unseen anomaly patterns or variations of expression (such as synonymous error messages), and are prone to missed or false alarms.
[0022] It lacks the intelligent ability to determine whether something is a genuine anomaly. Many logs (such as DEBUG and INFO level messages) contain words like "error" and "fail," but are not actually system failures. Current technology struggles to distinguish between "apparent anomalies" and "real failures," resulting in a large number of invalid alarms and increasing the burden on operations and maintenance.
[0023] Repair suggestions rely on human experience and have a low degree of automation. Even when anomalies are identified, existing systems typically only provide raw logs or simple classifications, failing to automatically generate actionable remediation steps. Maintenance personnel still need to manually consult documentation, community forums, or historical work orders, which is inefficient and prone to errors.
[0024] Knowledge bases are static and fixed, making continuous evolution difficult. Although some systems have introduced knowledge bases, they are mostly static rule bases maintained manually. These rule bases are updated late and cannot automatically learn from newly emerging exception logs and accumulate them into reusable knowledge, resulting in a stagnation in the system's intelligence level.
[0025] Therefore, to address the above reasons, a log anomaly diagnosis and repair method and system based on the collaboration of knowledge base and large model has been invented, aiming to solve the following technical problems: How to intelligently determine whether a log entry constitutes a real system anomaly, rather than relying solely on superficial keyword matching; How to automatically extract semantic-level fault features (such as problem attribution and initial repair direction) from massive logs without preset rules; How to build a dynamic log knowledge base that supports contextual association and causal reasoning, and realize the continuous learning and reuse of abnormal patterns; How can we integrate structured knowledge matching with large model reasoning capabilities to automatically generate accurate and actionable repair suggestions, thereby achieving end-to-end automatic diagnosis and repair of anomaly logs?
[0026] By introducing a multi-dimensional log pre-analysis mechanism, a knowledge base structure that supports pre-existing anomaly association, and a collaborative reasoning framework for large models and vector retrieval, this invention effectively overcomes the shortcomings of existing technologies in terms of accuracy, generalization, automation, and intelligence. Example 1
[0027] This embodiment provides a log anomaly diagnosis and repair method based on the collaboration of knowledge base and large model, such as... Figure 1 As shown, it includes: S1. Log collection and processing process: Obtain system operation logs from different sources, process and summarize them in a unified format.
[0028] S2. Log pre-analysis process: The logs are segmented, dimensionally split, and preliminarily analyzed.
[0029] S3. Log Knowledge Base Matching Process: Based on the preliminary analysis results, multi-dimensional matching is performed between the vector database and the log knowledge base. S4. Large Model Inference Process: Based on the matching results and the current log content, generate anomaly judgment and repair suggestions.
[0030] In one specific embodiment, the vector database may include vector representations of exception log examples, vector representations of exception log descriptions, and corresponding exception log IDs; the structure of the log knowledge base may include exception log IDs, exception log examples, exception repair steps, exception log descriptions, preceding exception log IDs, and preceding exception log descriptions.
[0031] like Figure 2 As shown, the log pre-analysis process specifically includes: Physical splitting of large files: splitting log files into multiple smaller files (GB-level log files can be split into fixed sizes, such as 100MB); Line-by-line reading and structuring: Reading line by line, the large language model is called to perform semantic parsing on each line of log, and the following structured dimensions are output: Source log: The original log text; Log pre-attribution problem: The large model summarizes the possible fault categories to which the log may be attributed based on its own knowledge (such as "database connection timeout", "memory overflow", etc.). Log preprocessing method: Preliminary suggested repair directions for large models; Previous error source log: Records the log content of the previous line that was judged to be an error, used for context association; Timestamp: Extracts time information from the log; if it is missing, it is marked as missing.
[0032] This structured data provides a multi-dimensional query basis for subsequent log knowledge base matching.
[0033] like Figure 3 As shown, the log knowledge base matching process specifically includes: The "log pre-attribution problem" is vectorized, and similar anomaly descriptions are retrieved from the vector database to return a list of IDs and the first similarity score. The "log pre-method" is vectorized, and similar repair descriptions are retrieved from the vector database to return a list of IDs and a second similarity score. Vectorize the "previous error source log", retrieve the preceding anomaly from the vector database, and return a list of IDs and a third similarity score; Combine three similarity scores (either by weighted average or by sorting) to generate a comprehensive matching ranking; Based on the Top-N matching results, retrieve the corresponding repair steps from the log knowledge base.
[0034] The large model inference process specifically includes: inputting the current log, the matched repair steps, and context information into the large model to generate the final diagnostic conclusion and executable repair suggestions.
[0035] The overall operation process can be as follows Figure 4 As shown: The log collector gathers logs and sends them to the aggregator; After the aggregator formats the logs, they are sent to the log pre-analysis module. The pre-analysis module segments the logs and analyzes them line by line to generate structured data; The log knowledge base matching module searches for the closest anomaly example in the knowledge base based on structured data; The large model inference module generates the final judgment and repair suggestions based on the matching results.
[0036] Based on the same inventive concept, this application also provides a system corresponding to the method in Embodiment 1, as detailed in Embodiment 2. Example 2
[0037] This embodiment provides a log anomaly diagnosis and repair system based on the collaboration of a knowledge base and a large model, such as... Figure 5 As shown, it includes: The log collection and processing module is used to acquire system operation logs from different sources, process and summarize them in a unified format; The log pre-analysis module is used to segment, dimensionally split, and perform preliminary analysis on logs. The log knowledge base matching module is used to perform multi-dimensional matching between the vector database and the log knowledge base based on the preliminary analysis results. The large model inference module is used to generate anomaly detection and repair suggestions based on the matching results and the current log content.
[0038] In one specific embodiment, the vector database includes vector representations of exception log examples, vector representations of exception log descriptions, and corresponding exception log IDs; the structure of the log knowledge base includes exception log IDs, exception log examples, exception repair steps, exception log descriptions, preceding exception log IDs, and preceding exception log descriptions.
[0039] The log pre-analysis module specifically includes: The splitting submodule is used to split log files into multiple smaller files; The structured submodule is used to perform semantic parsing on each line of log data using the large language model, outputting the following structured dimensions: Source log: The original log text; Log pre-attribution problem: Large models summarize the possible fault categories to which a log might belong based on their own knowledge; Log preprocessing method: Preliminary suggested repair directions for large models; Previous error source log: Records the log content of the previous line that was judged to be an error, used for context association; Timestamp: Extracts time information from the log; if it is missing, it is marked as missing.
[0040] like Figure 6 As shown, the log knowledge base matching module specifically includes: The first vectorization submodule is used to vectorize the "log pre-attribution problem", retrieve similar anomaly descriptions from the vector database, and return a list of IDs and the first similarity score; The second vectorization submodule is used to vectorize the "log pre-method", retrieve similar repair descriptions from the vector database, and return a list of IDs and a second similarity score. The third vectorization submodule is used to vectorize the "previous error source log", retrieve the previous anomaly from the vector database, and return a list of IDs and a third similarity score. The comprehensive ranking submodule is used to fuse the three similarity scores to generate a comprehensive matching ranking; The repair step retrieval submodule is used to retrieve the corresponding repair steps from the log knowledge base based on the Top-N matching results.
[0041] The large model inference module is specifically used to input the current log, the matched repair steps, and context information into the large model to generate the final diagnostic conclusion and executable repair suggestions.
[0042] Since the system described in Embodiment 2 of this invention is a system used to implement the method of Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this system based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All systems used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.
[0043] The technical solutions provided in the embodiments of the present invention have at least the following technical effects: 1. High degree of automation: The entire process, from log collection to generating remediation suggestions, is automated; 2. High analytical accuracy: Combining multi-dimensional matching and large-scale model reasoning improves the accuracy of anomaly identification; 3. High adaptability: The knowledge base can be dynamically updated to adapt to new and unusual scenarios; 4. High efficiency: The pre-analysis module segments logs, improving the processing efficiency of large models; 5. Good interpretability: Each analysis step has structured output, which facilitates traceability and optimization.
[0044] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0045] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0046] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.
[0047] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0048] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A log anomaly diagnosis and repair method based on knowledge base and large model collaboration, characterized in that, include: Log collection and processing process: Obtain system operation logs from different sources, process and summarize them in a unified format; Log pre-analysis process: Logs are segmented, dimensionally split, and preliminarily analyzed; Log knowledge base matching process: Based on the preliminary analysis results, multi-dimensional matching is performed between the vector database and the log knowledge base; Large model inference process: Based on the matching results and the current log content, generate anomaly judgment and repair suggestions.
2. The method according to claim 1, characterized in that: The vector database includes vector representations of exception log examples, vector representations of exception log descriptions, and corresponding exception log IDs; the structure of the log knowledge base includes exception log IDs, exception log examples, exception repair steps, exception log descriptions, preceding exception log IDs, and preceding exception log descriptions.
3. The method according to claim 1, characterized in that: The log pre-analysis process specifically includes: splitting the log file into multiple smaller files, performing semantic parsing on each line of the log using a large language model, and outputting the following structured dimensions: Source log: The original log text; Log pre-attribution problem: Large models summarize the possible fault categories to which a log might belong based on their own knowledge; Log preprocessing method: Preliminary suggested repair directions for large models; Previous error source log: Records the log content of the previous line that was judged to be an error, used for context association; Timestamp: Extracts time information from the log; if it is missing, it is marked as missing.
4. The method according to claim 2, characterized in that: The log knowledge base matching process specifically includes: The "log pre-attribution problem" is vectorized, and similar anomaly descriptions are retrieved from the vector database to return a list of IDs and the first similarity score. The "log pre-method" is vectorized, and similar repair descriptions are retrieved from the vector database to return a list of IDs and a second similarity score. Vectorize the "previous error source log", retrieve the preceding anomaly from the vector database, and return a list of IDs and a third similarity score; By combining the three similarity scores, a comprehensive matching ranking is generated; Based on the Top-N matching results, retrieve the corresponding repair steps from the log knowledge base.
5. The method according to claim 4, characterized in that, The large model reasoning process specifically includes: inputting the current log, the matched repair steps, and context information into the large model to generate the final diagnostic conclusion and executable repair suggestions.
6. A log anomaly diagnosis and repair system based on knowledge base and large model collaboration, characterized in that, include: The log collection and processing module is used to acquire system operation logs from different sources, process and summarize them in a unified format; The log pre-analysis module is used to segment, dimensionally split, and perform preliminary analysis on logs. The log knowledge base matching module is used to perform multi-dimensional matching between the vector database and the log knowledge base based on the preliminary analysis results. The large model inference module is used to generate anomaly detection and repair suggestions based on the matching results and the current log content.
7. The system according to claim 6, characterized in that: The vector database includes vector representations of exception log examples, vector representations of exception log descriptions, and corresponding exception log IDs; the structure of the log knowledge base includes exception log IDs, exception log examples, exception repair steps, exception log descriptions, preceding exception log IDs, and preceding exception log descriptions.
8. The system according to claim 6, characterized in that, The log pre-analysis module includes: The splitting submodule is used to split log files into multiple smaller files; The structured submodule is used to perform semantic parsing on each line of log data using the large language model, outputting the following structured dimensions: Source log: The original log text; Log pre-attribution problem: Large models summarize the possible fault categories to which a log might belong based on their own knowledge; Log preprocessing method: Preliminary suggested repair directions for large models; Previous error source log: Records the log content of the previous line that was judged to be an error, used for context association; Timestamp: Extracts time information from the log; if it is missing, it is marked as missing.
9. The system according to claim 8, characterized in that: The log knowledge base matching module specifically includes: The first vectorization submodule is used to vectorize the "log pre-attribution problem", retrieve similar anomaly descriptions from the vector database, and return a list of IDs and the first similarity score; The second vectorization submodule is used to vectorize the "log pre-method", retrieve similar repair descriptions from the vector database, and return a list of IDs and a second similarity score. The third vectorization submodule is used to vectorize the "previous error source log", retrieve the previous anomaly from the vector database, and return a list of IDs and a third similarity score. The comprehensive ranking submodule is used to fuse the three similarity scores to generate a comprehensive matching ranking; The repair step retrieval submodule is used to retrieve the corresponding repair steps from the log knowledge base based on the Top-N matching results.
10. The system according to claim 9, characterized in that, The large model inference module is specifically used to input the current log, the matched repair steps, and context information into the large model to generate the final diagnostic conclusion and executable repair suggestions.