A fault diagnosis method and system based on multi-modal fusion and knowledge graph

Abstract

This invention discloses a fault diagnosis method and system based on multimodal fusion and knowledge graph, relating to the field of information technology operation and maintenance. The method includes: collecting and fusing multimodal operation and maintenance data; inputting the fused data into a dedicated model layer to calculate object feature attention and filter out a set of highly relevant object features; retrieving matching historical fault patterns from a vectorized document library and fault pattern knowledge graph of a knowledge factory based on this feature set; combining natural language questions, object feature sets, and retrieved fault patterns into a prompt template, inputting it into a large language model to generate diagnostic results, and feeding back the diagnostic process and results to the knowledge factory to update the fault pattern knowledge graph. This invention significantly improves the accuracy and efficiency of fault diagnosis, shortens fault recovery time, and constructs a self-optimizing, increasingly intelligent operation and maintenance knowledge system, providing strong technical support for the intelligent operation and maintenance of complex information systems.

Description

Technical Field

[0001] This invention relates to the technical field of information technology operation and maintenance, and more specifically to a fault diagnosis method and system based on multimodal fusion and knowledge graph. Background Technology

[0002] Currently, with the increasing complexity of enterprise information systems and the ever-increasing demands for business continuity, operation and maintenance (O&M) systems are facing severe challenges. While existing O&M tools have achieved some success in monitoring and alerting, they still have significant shortcomings in automation and intelligence, specifically in the following aspects:

[0003] Inefficient fault location and difficulty in root cause analysis: Existing operations and maintenance (O&M) systems lack effective intelligent analysis and correlation capabilities when faced with massive and heterogeneous O&M data, such as logs, metrics, and alarms. When a fault occurs, O&M personnel struggle to quickly and accurately locate the root cause from this sea of ​​data, resulting in excessively long Mean Time To Repair (MTTR), severely restricting business continuity and stability. Traditional rule-based monitoring methods are severely inadequate in terms of intelligent analysis and autonomous decision-making capabilities when dealing with complex, sudden, and unknown faults.

[0004] Key performance bottlenecks are prominent, and optimization heavily relies on expert experience: Some critical business systems frequently experience slowdowns or even instance restarts due to high concurrency and heavy load. Especially in complex database business scenarios, tracing the source of problematic SQL queries is difficult. Their complex logical structures and execution plans make performance optimization work highly dependent on the personal experience of senior operations and maintenance personnel, lacking systematic and automated analysis and optimization methods.

[0005] The value of operational data is not fully realized, and intelligent decision-making capabilities are lacking: With the explosive growth of data volume, the multi-source heterogeneous data generated during the operation and maintenance process, such as CMDB, monitoring indicators, logs, and historical work orders, is difficult to deeply mine and effectively utilize manually. Although some enterprises have begun to explore technologies such as knowledge graphs, the overall application is relatively fragmented, failing to form an effective linkage and synergy from "data" to "knowledge" to "intelligent decision-making." This results in the operation and maintenance system lacking the ability to assist in decision-making and automate handling, becoming a key bottleneck in improving operation and maintenance efficiency and service quality.

[0006] Therefore, there is an urgent need for a fault diagnosis method and system based on multimodal fusion and knowledge graph to break through the limitations of traditional operation and maintenance rules and solve the core pain points of complex systems in the operation and maintenance process. Summary of the Invention

[0007] In view of the shortcomings of the existing technology, the present invention provides a fault diagnosis method and system based on multimodal fusion and knowledge graph, which aims to solve the problems of weak multimodal data processing capability, poor diagnosis effect of single engine and insufficient dual-engine collaboration of existing fault diagnosis technology, and improve the accuracy and efficiency of fault diagnosis of complex systems.

[0008] The technical solution of the present invention is achieved in the following ways: In a first aspect, embodiments of the present invention provide a fault diagnosis method based on multimodal fusion and knowledge graph, comprising the following steps: S1. Collect multimodal operation and maintenance data during the operation of IT infrastructure, perform data cleaning, time alignment and feature extraction to form fused data; S2. Input the fused data into the dedicated model layer, calculate the attention of each object feature in the fault diagnosis task, and filter out the set of highly concerned object features based on the attention ranking algorithm. S3. Based on the object feature set, retrieve relevant private domain knowledge from the knowledge factory, and match the object feature set and the private domain knowledge using graph traversal algorithm and vector similarity to obtain the historical fault mode that best matches the current fault features. S4. Combine the natural language question input by the user, the object feature set, and the retrieved fault mode into a prompt template, input it into the large language model, generate fault diagnosis results, receive confirmation or correction information from the user regarding the fault diagnosis results, combine the confirmed fault mode, fault description, root cause description, and repair plan into a four-tuple structure, extract its feature vector, and feed it back to the knowledge factory to update the fault mode knowledge graph.

[0009] Furthermore, in step S1, The IT infrastructure includes: hardware, operating system, database, and middleware; The multimodal operation and maintenance data includes: CMDB, indicator system, logs, work orders, alarms, operation, maintenance, management and security knowledge.

[0010] Furthermore, in step S2, the formula for the attention ranking algorithm is as follows:

[0011] In the formula, Aself(f i ,t) is a feature f i At time t, based on the degree of anomaly in its own historical changes, Adist(f) i ,t) is a feature f i The relative attention given to similar objects in the task domain, Aprior(f) i) represents the prior attention calculated based on a preset threshold, where α, β, and γ are weighting coefficients that satisfy α+β+γ=1.

[0012] Furthermore, in step S2, the dedicated model layer includes at least one of a machine learning model, a deep learning model, and a statistical algorithm.

[0013] Furthermore, in step S3, when obtaining the historical fault mode that best matches the current fault characteristics, the following similarity calculation formula is used:

[0014] In the formula, Q is the current fault feature vector, and P k For the k-th historical fault mode, Cosine(V) Q V Pk ) represents the cosine similarity of vectorized features, GraphDist(G Q G Pk ) represents the graph distance similarity of the fault propagation path, and λ is the fusion weight coefficient.

[0015] Furthermore, in step S3, the knowledge factory includes: a vectorized document library and a fault mode knowledge graph.

[0016] Furthermore, in step S3, the construction of the fault mode knowledge graph includes: Extract entities and relationships between entities from multimodal operation and maintenance data, specifically including applications, databases, servers, middleware, fault phenomena, root causes of faults, and remediation solutions; A graph neural network is used to embed entities and relationships to establish a fault propagation path; Through the knowledge graph adaptive update mechanism, new fault modes confirmed each time are dynamically added to the knowledge graph.

[0017] Furthermore, step S4 specifically includes: The preset prompt template is invoked, and the user's input natural language question, the highly concerned object feature set, and the retrieved fault mode are filled into the corresponding positions in the template; The filled-in prompt template is input into the large language model, which can be an open-source model or a commercial model. Receive the generated results returned by the large language model, which include root cause analysis, repair suggestions, and repair scripts.

[0018] Furthermore, the diagnostic results output and knowledge feedback include: Output the generated fault diagnosis results to the user interface or API interface; Receive user confirmation or correction information regarding diagnostic results as feedback data; The confirmed fault modes, fault descriptions, root cause descriptions, and remediation solutions are updated to the fault mode knowledge graph to achieve continuous knowledge evolution.

[0019] Furthermore, it also includes SQL intelligent optimization tasks and inspection intelligent analysis tasks; The SQL intelligent optimization task will automatically analyze TopSQL and generate optimization suggestions; Secondly, embodiments of the present invention provide a fault diagnosis system based on multimodal fusion and knowledge graph, applying the method described in the first aspect embodiment, the system comprising: Feature extraction module: used to acquire multimodal fault input information, and to parse and process the multimodal fault input information to extract key fault features; Knowledge graph reasoning module: used to call the knowledge graph reasoning engine, perform graph reasoning on the key fault features based on the fault knowledge graph, and obtain structured reasoning results; The retrieval enhancement semantic analysis module is used to call the retrieval enhancement generation engine, use the structured reasoning results as the query context, retrieve relevant knowledge from the vectorized knowledge base, and perform semantic analysis in conjunction with the large language model to obtain the semantic analysis results. Diagnostic fusion output module: used to fuse the structured reasoning results and the semantic analysis results to output a fault diagnosis conclusion.

[0020] Compared with the prior art, the present invention has the following core beneficial effects: Significantly improving the accuracy and efficiency of fault diagnosis and shortening fault recovery time: This invention constructs a comprehensive operational data foundation by collecting and integrating multimodal operational data such as CMDB, metrics, logs, and work orders. Utilizing a dedicated model layer to calculate and sort the attention levels of massive amounts of data, it can quickly filter out the feature sets of highly relevant objects of interest to the fault, effectively avoiding data overload and information noise. Combining the vectorized document library and fault mode knowledge graph in the knowledge factory, through graph traversal and vector similarity matching, it accurately retrieves the most matching historical fault patterns, providing high-quality, highly relevant contextual information for the large model. This process transforms traditional manual experience-driven fault investigation into data- and knowledge-driven intelligent diagnosis, significantly improving the accuracy and speed of root cause localization, and ultimately achieving a significant reduction in mean time to repair (MTTR).

[0021] This invention enables intelligent optimization of database performance bottlenecks, reducing reliance on expert experience. It extends beyond fault diagnosis to intelligent SQL optimization tasks. By automatically analyzing TopSQL and combining performance optimization knowledge from a knowledge factory, it accurately identifies SQL statements causing performance bottlenecks and automatically generates optimization suggestions and scripts. This process utilizes specialized models to analyze SQL characteristics, such as table selectivity and data distribution, replacing traditional manual analysis and optimization methods. It solves the problems of tracing the source of problematic SQL in complex business scenarios and the high dependence of optimization on human experience. By optimizing typical TopSQL queries, database query response time can be effectively reduced, significantly improving database throughput and concurrency capabilities.

[0022] Building an automated operation and maintenance closed loop enables continuous evolution of operation and maintenance knowledge: The knowledge factory in this invention supports version management and adaptive updates of the fault mode library. After each fault diagnosis, the system dynamically updates the fault mode knowledge graph with the diagnosis process, the confirmed fault mode, and the corresponding repair solution. This mechanism allows operation and maintenance knowledge to accumulate and evolve continuously with system use, achieving automated construction and continuous improvement of the knowledge base. This avoids the inefficient model of relying entirely on manual maintenance for knowledge accumulation, thereby continuously improving the system's intelligent processing capabilities and enabling a sustained increase in the efficiency of handling typical faults.

[0023] This invention reduces reliance on large, general-purpose models, enhancing system usability and responsiveness. By preprocessing and feature-selecting real-time data through a dedicated model layer and combining this with precise retrieval from a knowledge factory, it provides large models with refined, task-relevant information, significantly reducing the demands on the model's parameterized knowledge and complex reasoning capabilities. The system can then utilize smaller, more efficient models to perform specialized operational tasks. This not only reduces the cost of model deployment and inference but also improves the system's responsiveness and accuracy for specific tasks, making this technical solution more practical and valuable for widespread adoption in real-world production environments. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0025] Figure 1 This is a flowchart of a fault diagnosis method based on multimodal fusion and knowledge graph provided in an embodiment of the present invention.

[0026] Figure 2This is a flowchart illustrating the internal construction and use of a knowledge factory as provided in an embodiment of the present invention.

[0027] Figure 3 This is a flowchart comparing the overall process provided in the embodiments of the present invention.

[0028] Figure 4 This is a flowchart of a fault diagnosis system based on multimodal fusion and knowledge graph provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Example 1: like Figure 1 As shown in the figure, this invention discloses a fault diagnosis method based on multimodal fusion and knowledge graph, including the following steps: S1. Collect multimodal operation and maintenance data during the operation of IT infrastructure, perform data cleaning, time alignment and feature extraction to form fused data; Specifically, the system first collects multimodal operation and maintenance data from multiple data sources within the information system. This includes, but is not limited to: resource and topology relationship data from the Configuration Management Database (CMDB), time-series data from the performance monitoring indicator system, application and system logs, historical fault work orders, alarm event records, and knowledge data related to operation, maintenance, management, and security. These data come from diverse sources and have varying formats; directly using them for analysis would lead to information redundancy and time misalignment. Therefore, the system first performs data cleaning, removing outliers and duplicate data. Then, it uses timestamp alignment technology to unify data from different sources into the same time window. Finally, it uses feature extraction methods to extract key information from the cleaned and aligned data, forming structured fused data, laying the data foundation for subsequent intelligent analysis.

[0031] S2. Input the fused data into the dedicated model layer, calculate the attention of each object feature in the fault diagnosis task, and filter out the set of highly concerned object features based on the attention ranking algorithm. Specifically, the system inputs the fused data generated in step S1 into a dedicated model layer. This model layer integrates multiple algorithms, including machine learning models, deep learning models, statistical algorithms, and database expert experience rules. Its core task is to calculate the "attention level" of each object feature in the fault diagnosis task. The calculation of attention level combines three dimensions: first, the degree of anomaly based on the historical trend of the feature itself; second, the distribution characteristics of the feature in the set of similar objects in the current task domain; and third, prior knowledge based on preset thresholds or expert experience. Subsequently, the system comprehensively ranks all features using an attention level ranking algorithm, selecting the top-ranked set of object features. This feature set accurately identifies the most likely associated key objects and their abnormal behaviors in the current fault scenario, effectively reducing the input scale of subsequent analysis and improving the efficiency and accuracy of diagnosis.

[0032] S3. Based on the object feature set, retrieve relevant private domain knowledge from the knowledge factory. The knowledge factory includes a vectorized document library and a fault mode knowledge graph. Through graph traversal algorithm and vector similarity matching, obtain the historical fault mode that best matches the current fault feature. Specifically, the system uses the highly relevant object feature set selected in step S2 as query conditions to retrieve relevant private domain knowledge from the "knowledge factory." The knowledge factory maintains two core knowledge bases: one is a vectorized document library, which stores unstructured knowledge such as operation and maintenance manuals, technical documents, and historical work order processing records, and is retrieved through vector similarity matching; the other is a fault mode knowledge graph, which stores historical fault entities in a graph structure, such as applications, databases, servers, fault phenomena, root causes, and their relationships. The system uses a graph traversal algorithm to find matching historical fault subgraphs along the fault propagation path, and combines the vector similarity matching results to calculate the comprehensive similarity between the current feature set and historical fault modes, thereby obtaining one or more historical fault modes that best match the current fault features. This process enables the intelligent reuse of past operation and maintenance experience.

[0033] S4. Combine the natural language question input by the user, the object feature set, and the retrieved fault mode into a prompt template, input it into the large language model, generate the fault diagnosis result, and form the diagnosis process and result into a four-tuple structure, extract its feature vector, and feed it back to the knowledge factory to update the fault mode knowledge graph.

[0034] Specifically, the system first calls a preset prompt template, structurally combining the user's input natural language question (e.g., "The database response is slow"), the object feature set obtained in step S2, and the historical fault patterns retrieved in step S3 to form a high-quality prompt template. This template is then input into a large language model, which utilizes its powerful reasoning and generation capabilities, combined with the contextual information in the prompt, to generate the final fault diagnosis result. The diagnosis result typically includes root cause analysis, specific repair suggestions, and executable repair scripts. Furthermore, the system sends key information from this diagnosis, such as input features, matched fault patterns, and user confirmation or correction of the diagnosis result, as feedback to the knowledge factory. Based on this, the knowledge factory dynamically updates the fault pattern knowledge graph, assembling the confirmed new fault pattern, fault description, root cause description, and repair plan into a four-tuple structure, extracting its feature vectors, and adding them to the graph. This enables continuous accumulation and evolution of operational knowledge, making the system increasingly intelligent with use.

[0035] This invention effectively solves the problems of low efficiency, reliance on human experience, and difficulty in knowledge accumulation in fault diagnosis faced by traditional operation and maintenance systems by constructing a complete technical closed loop from data collection, feature screening, knowledge retrieval to result generation and knowledge feedback.

[0036] This method first integrates multi-source heterogeneous operation and maintenance data to provide a comprehensive and orderly data foundation for analysis. Second, it uses a dedicated model layer to intelligently filter massive amounts of data, accurately locate key features, and significantly reduce the complexity of subsequent analysis. Next, it uses a vectorized document library and fault mode knowledge graph in the knowledge factory to achieve structured storage and intelligent reuse of historical operation and maintenance experience. Finally, it combines the generation capabilities of a large language model to output interpretable diagnostic results and repair solutions, and achieves continuous knowledge evolution through a feedback mechanism.

[0037] The entire technical solution deeply integrates "data-driven" and "knowledge-driven" approaches, significantly improving the accuracy and efficiency of fault diagnosis, shortening fault recovery time, and building a self-optimizing, increasingly intelligent operation and maintenance knowledge system, providing strong technical support for the intelligent operation and maintenance of complex information systems.

[0038] The following describes in detail the fault diagnosis method based on multimodal fusion and knowledge graph provided by the embodiments of the present invention, which specifically includes the following steps: Step S1: Collect multimodal operation and maintenance data during the operation of IT infrastructure, perform data cleaning, time alignment, and feature extraction to form fused data, including: Specifically, this step first collects multimodal operation and maintenance data from multiple data sources within the information system. The IT infrastructure includes hardware, operating systems, databases, and middleware; the multimodal operation and maintenance data includes, but is not limited to: resource and topology relationship data from the Configuration Management Database (CMDB), time-series data from performance monitoring metrics systems such as CPU utilization, memory usage, disk I / O, and network traffic, application and system logs, historical fault tickets, alarm event records, and knowledge data related to operation, maintenance, management, and security. These data come from diverse sources and are in various formats; directly using them for analysis would lead to information redundancy and time misalignment.

[0039] To this end, the system first performs data cleaning, using a rule engine combined with scripts to remove outliers and duplicate data, and to complete missing data. For example, for the same alarm generated by the same host within a 5-minute time window, only the first record is retained, along with its frequency. Subsequently, timestamp alignment technology is used to unify data from different sources into the same time window, ensuring time-series consistency. Finally, feature extraction methods are used to extract key information from the cleaned and aligned data. For example, statistical algorithms are used to extract time-series and statistical features such as mean, variance, and peak value from structured data, and pre-trained language models are used to extract semantic features from unstructured text. These features are then fused to form structured fused data, laying the data foundation for subsequent intelligent analysis.

[0040] Preferably, multimodal operation and maintenance data specifically includes: CMDB, indicator system, logs, work orders, alarms, operation, maintenance, operation and security knowledge. These data together constitute the complete data foundation for operation and maintenance analysis.

[0041] Step S2: Input the fused data into the dedicated model layer, calculate the attention of each object feature in the fault diagnosis task, and filter out the set of highly attentionable object features based on the attention ranking algorithm; Specifically, the system inputs the fused data generated in step S1 into a dedicated model layer. This model layer integrates various algorithms, including machine learning models such as random forests and support vector machines, deep learning models such as LSTM and Transformers, statistical algorithms such as time series decomposition, anomaly detection algorithms, and database expert experience rules. Its core task is to calculate the "attention level" of each object feature, such as "CPU utilization of database A" and "response time of application B," in the fault diagnosis task.

[0042] The formula for the attention ranking algorithm is as follows:

[0043] In the formula, Aself(fi,t) represents the degree of anomaly of feature fi at time t based on its own historical changes. For example, it can be determined by calculating the deviation multiple between the current value and the historical baseline (such as the average of the same time in the previous 7 days). Adist(fi,t) represents the relative degree of attention of feature fi in the distribution of similar objects in the task domain. For example, when comparing the CPU utilization of all instances in a database cluster, after Z-score normalization, the instance with the larger deviation from the mean scores higher. Aprior(fi) represents the prior degree of attention based on a preset threshold or expert experience. For example, when the CPU utilization is higher than 95%, this item is scored as 1. α, β, and γ are configurable weight coefficients that satisfy α+β+γ=1. In practical applications, they can be optimized according to different task scenarios. For example, for sudden increase faults, α can be set to a larger value; for resource contention faults, β can be set to a larger value.

[0044] Preferably, Aself(fi,t) can separate the trend, periodic and residual components through a time series decomposition model, and calculate the attention level based on the anomaly of the residual components; Adist(fi,t) can be determined by calculating the percentile of the feature value of this feature value among all similar objects in the same task domain; Aprior(fi) is obtained by looking up a table according to the threshold rule table preset by experts.

[0045] Subsequently, the system uses a focus ranking algorithm to comprehensively rank all features and select the top-ranked object feature set. This ranking algorithm weights the global focus ranking of each object feature with its local ranking within the object type to obtain the final ranking. This approach retains high-focus object features while also considering the diversity of object features, avoiding suboptimal solutions. This feature set accurately identifies the most likely associated key objects and their abnormal behaviors in the current fault scenario, effectively reducing the input size for subsequent analysis and improving the efficiency and accuracy of diagnosis.

[0046] Step S3: Based on the object feature set, retrieve relevant private domain knowledge from the knowledge factory. The knowledge factory includes a vectorized document library and a fault mode knowledge graph. Through graph traversal algorithm and vector similarity matching, obtain the historical fault mode that best matches the current fault features.

[0047] The system uses the set of highly relevant object features selected in step S2 as query conditions to retrieve relevant private domain knowledge from the "knowledge factory." For example... Figure 2As shown, the internal construction and usage process of the knowledge factory specifically includes: First, preprocessing, splitting, vectorizing, and indexing professional knowledge files such as PDF, Word, and TXT to form a vectorized knowledge base; when the user inputs a question, the system vectorizes the question, retrieves it from the knowledge base, obtains the Top K relevant paragraphs, then calls the prompt engineering template to combine the user's question with the knowledge base paragraphs to form a context, and finally calls the large language model to generate intelligent question answering results.

[0048] In this step, the knowledge factory specifically maintains two types of core knowledge bases: Vectorized document library: This library stores unstructured knowledge such as operation and maintenance manuals, technical documents, and historical work order processing records. The system converts the current fault feature set into a vector representation and retrieves relevant knowledge fragments with Top-K similarity from the vectorized knowledge library using cosine similarity matching.

[0049] Fault Mode Knowledge Graph: It stores historical fault entities in graph structure, such as applications, databases, servers, fault symptoms, root causes, remediation solutions, and their relationships such as "caused," "solution," "impact," and "dependencies." The system uses graph traversal algorithms, such as random walk with restart, to search for matching historical fault subgraphs along the fault propagation path.

[0050] Preferably, the fault mode knowledge graph supports fault mode library management, specifically including: mode management, which supports adding, deleting, querying and modifying fault mode information; fault mode knowledge library query, which supports obtaining fault mode knowledge library information; and fault mode version management, which supports comparing, merging and rolling back fault mode versions.

[0051] Preferably, the construction of the fault mode knowledge graph includes: extracting entities and relationships between entities from multimodal operation and maintenance data, specifically including applications, databases, servers, middleware, fault phenomena, root causes of faults, and repair solutions; using graph neural networks, such as GCN and GAT, to embed entities and relationships and establish fault propagation paths; and dynamically adding new fault modes confirmed by each diagnosis to the knowledge graph through an adaptive update mechanism.

[0052] When obtaining the historical fault modes that best match the current fault characteristics, the following similarity calculation formula is used:

[0053] In the formula, Q is the current fault feature vector, and P k For the k-th historical fault mode, Cosine(V) Q V Pk ) represents the cosine similarity of vectorized features, GraphDist(GQ G Pk ) represents the graph distance similarity of the fault propagation path, and λ is the fusion weight coefficient.

[0054] For example, it can be measured by calculating the edit distance or node overlap between two faulty subgraphs, where λ is the fusion weight coefficient used to balance the contributions of the two similarities.

[0055] By calculating the comprehensive similarity, the system can obtain one or more historical fault modes that best match the current fault characteristics. This process enables the intelligent reuse of past operation and maintenance experience.

[0056] Step S4: Combine the natural language question input by the user, the object feature set, and the retrieved fault mode into a prompt template, input it into the large language model, generate the fault diagnosis result, and feed back the diagnosis process and result to the knowledge factory to update the fault mode knowledge graph. Specifically, the system first calls a preset prompt template, and structurally combines the user-input natural language question, such as "The database response is slow," the object feature set obtained in step S2, such as "The CPU utilization of database A is as high as 98%" and "The lock wait time for table X is too long," and the historical failure patterns retrieved in step S3 to form a high-quality prompt template. This template is designed to guide the large model in step-by-step reasoning, for example: "You are a senior database operations and maintenance expert."

[0057] The current user's problem is: [User Problem].

[0058] The following are the most relevant anomaly features detected by the system in real time: [Object Feature Set].

[0059] Based on these characteristics, the following best-matching fault modes were found in the historical fault database: [Fault modes retrieved].

[0060] Based on the above information, please analyze the root cause of the failure, provide specific repair suggestions, and an executable repair script. The template is then input into a large language model, such as the open-source Qwen series models or the commercial GPT series models. The large model, leveraging its powerful reasoning and generation capabilities and combined with the contextual information in the prompts, generates the final fault diagnosis result. The diagnosis result typically includes root cause analysis, specific repair suggestions, and executable repair scripts.

[0061] Furthermore, the application layer of the system of this invention also includes a general task agent, which includes an API agent module for performing intent recognition, target API selection, API call sequence planning, and API call execution. During fault diagnosis, the general task agent can automatically call relevant API interfaces based on the diagnosis results to perform preset repair operations, such as restarting services, adjusting parameters, and executing SQL optimization scripts, thereby automating the handling of operation and maintenance tasks.

[0062] Finally, the system sends key information from this diagnosis, such as input features, matched fault modes, and user confirmation or correction of the diagnostic results, as feedback to the knowledge factory. Based on this, the knowledge factory dynamically updates the fault mode knowledge graph, adding confirmed new fault modes, remedial solutions, and their feature vectors to the graph, achieving continuous knowledge evolution. Specifically, the diagnostic result output and knowledge feedback include: outputting the generated fault diagnosis results to the user interface or API interface; receiving user confirmation or correction information for the diagnostic results as feedback data; and updating the confirmed fault modes, fault descriptions, root cause descriptions, and remedial solutions to the fault mode knowledge graph, achieving continuous knowledge evolution.

[0063] The system of this invention uses MoTAServ for data preprocessing and index building, and DB-GPT for retrieval and generation. The front-end and router services are responsible for the basic logic of RAQ. For inspection reports, private knowledge bases, and specified document URLs, the system performs intelligent analysis and RAQ question answering. The back-end service interacts with vllm / sglang through ProxyLLM to implement Qwen large-scale model inference services. The API agent is responsible for intent recognition, target API selection, API call sequence planning, API calls, and question responses, thereby achieving functions such as SQL optimization and alarm diagnosis.

[0064] The overall process of this invention will now be compared with that of conventional methods: like Figure 3 As shown, in traditional methods, when specific operational tasks need to be completed, such as fault diagnosis and SQL optimization, the system directly inputs unfiltered, full-volume private domain knowledge into the large language model. This full-volume private domain knowledge includes: database and runtime environment data, basic resource information, real-time monitoring data, vectorized documents, knowledge graphs, etc., and the amount of data is extremely large.

[0065] This direct input method has the following technical problems: First, the sheer volume of data in the entire knowledge base causes the attention mechanism of large models to become scattered, making it impossible to focus on key information. Second, large models themselves cannot accurately analyze specialized data formats such as time series data, and additional data preprocessing is required. Third, a large number of token inputs leads to slow response speed of large models, and the upper limit of the number of tokens required for large models is relatively high, which increases the overhead of computing resources.

[0066] The aforementioned problems collectively result in low accuracy and low resource efficiency of traditional methods, making it difficult to meet the requirements of production environments for efficiency and accuracy in fault diagnosis.

[0067] To address the aforementioned issues, this invention innovatively adds a "real-time data analysis dedicated model layer" before the large data input model. This dedicated model layer integrates multiple algorithms, including machine learning models, deep learning models, statistical algorithms, deep search engines, and intelligent mini-models, to intelligently filter and select all private domain knowledge. Specifically, this dedicated model layer uses an attention calculation algorithm to extract "objects and features requiring high attention" from massive amounts of real-time data and general knowledge. These selected key information data are small in volume, have obvious features, and are highly relevant to the current task. Subsequently, the system combines this refined key information with specific task prompt templates and inputs it into the large language model for analysis and reasoning.

[0068] The technical advantages of the method of this invention are reflected in: First, based on the task, the highly relevant knowledge required for the task is accurately obtained from comprehensive private domain knowledge, especially real-time knowledge. This ensures that the large model can analyze and complete the task based on relatively comprehensive key information of the real environment, so that even a smaller model can achieve better results. Second, the number of tokens for knowledge that requires close attention is relatively small, which can effectively improve the response speed of large models and significantly reduce the consumption of computing resources. Third, by using specialized models to pre-analyze specialized data formats such as time series data, the shortcomings of large models in specialized data analysis capabilities are compensated for.

[0069] In summary, this technical approach enables a collaborative division of labor where a dedicated model handles data filtering and feature extraction, while a general model handles inference and generation. This approach significantly improves the system's response efficiency and resource utilization while ensuring diagnostic accuracy.

[0070] Example 2: like Figure 4 As shown, based on the same inventive concept, this embodiment of the invention also provides a fault diagnosis system based on multimodal fusion and knowledge graph, the system comprising: Interaction layer: Used to provide various database status queries and statistics, various database operation and maintenance operations, database fault detection and handling, and database knowledge Q&A functions; Application layer: includes dedicated task intelligence agents and general task intelligence agents. The dedicated task intelligence agents include SQL optimization modules, inspection and analysis modules, and fault diagnosis modules, which are used to execute corresponding intelligent operation and maintenance tasks. The general task intelligence agents are used to execute various multi-step operation and maintenance tasks and resource monitoring information statistics tasks. Model service layer: includes general model and special model. The general model is a large language model. The special model includes machine learning model, deep learning model, statistical algorithm and special small model, which are used to calculate and rank attention of multimodal data and output a set of highly concerned object features. Data Layer: Used to store real-time private domain knowledge and general private domain knowledge. The real-time private domain knowledge includes basic resource information, time-series monitoring data, and object attribute information. The general private domain knowledge includes vectorized documents and knowledge graphs, and supports vectorized retrieval, graph query, relational data query, and time-series data query functions.

[0071] Since these systems and the principles underlying the problems they solve are similar to the aforementioned fault diagnosis methods based on multimodal fusion and knowledge graphs, the implementation of this system can refer to the implementation of the aforementioned methods, and the repetitions will not be repeated.

[0072] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0073] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A fault diagnosis method based on multimodal fusion and knowledge graph, characterized in that, Includes the following steps: S1. Collect multimodal operation and maintenance data during the operation of IT infrastructure, perform data cleaning, time alignment and feature extraction to form fused data; S2. Input the fused data into the dedicated model layer, calculate the attention of each object feature in the fault diagnosis task, and filter out the set of highly concerned object features based on the attention ranking algorithm. S3. Based on the object feature set, retrieve relevant private domain knowledge from the knowledge factory, and match the object feature set and the private domain knowledge using graph traversal algorithm and vector similarity to obtain the historical fault mode that best matches the current fault features. S4. Combine the natural language question input by the user, the object feature set, and the retrieved fault mode into a prompt template, input it into the large language model, generate fault diagnosis results, receive confirmation or correction information from the user regarding the fault diagnosis results, combine the confirmed fault mode, fault description, root cause description, and repair plan into a four-tuple structure, extract its feature vector, and feed it back to the knowledge factory to update the fault mode knowledge graph.

2. The method as described in claim 1, characterized in that, In step S1 The IT infrastructure includes: hardware, operating system, database, and middleware; The multimodal operation and maintenance data includes: CMDB, indicator system, logs, work orders, alarms, operation, maintenance, management and security knowledge.

3. The method as described in claim 1, characterized in that, In step S2, the formula for the attention ranking algorithm is as follows: In the formula, Aself(f i ,t) is a feature f i At time t, based on the degree of anomaly in its own historical changes, Adist(f) i ,t) is a feature f i The relative attention given to similar objects in the task domain, Aprior(f) i ) represents the prior attention calculated based on a preset threshold, where α, β, and γ are weighting coefficients that satisfy α+β+γ=1.

4. The method as described in claim 1, characterized in that, In step S2, the dedicated model layer includes at least one of machine learning models, deep learning models, and statistical algorithms.

5. The method as described in claim 1, characterized in that, In step S3, when obtaining the historical fault mode that best matches the current fault characteristics, the following similarity calculation formula is used: In the formula, Q is the current fault feature vector, and P k For the k-th historical fault mode, Cosine(V) Q V Pk ) represents the cosine similarity of vectorized features, GraphDist(G Q G Pk ) represents the graph distance similarity of the fault propagation path, and λ is the fusion weight coefficient.

6. The method as described in claim 1, characterized in that, In step S3, the knowledge factory includes a vectorized document library and a fault mode knowledge graph.

7. The method as described in claim 6, characterized in that, The construction of the fault mode knowledge graph includes: Extract entities and relationships between entities from multimodal operation and maintenance data, specifically including applications, databases, servers, middleware, fault phenomena, root causes of faults, and remediation solutions; A graph neural network is used to embed entities and relationships to establish a fault propagation path; Through the knowledge graph adaptive update mechanism, new fault modes confirmed each time are dynamically added to the knowledge graph.

8. The method as described in claim 1, characterized in that, Step S4 specifically includes: The preset prompt template is invoked, and the user's input natural language question, the highly concerned object feature set, and the retrieved fault mode are filled into the corresponding positions in the template; The filled-in prompt template is input into the large language model, which can be an open-source model or a commercial model. Receive the generated results returned by the large language model, which include root cause analysis, repair suggestions, and repair scripts.

9. The method as described in claim 1, characterized in that, In step S4, the diagnostic result output and knowledge feedback include: Output the generated fault diagnosis results to the user interface or API interface; Receive user confirmation or correction information regarding diagnostic results as feedback data; The confirmed fault modes, fault descriptions, root cause descriptions, and remediation solutions are updated to the fault mode knowledge graph to achieve continuous knowledge evolution.

10. A fault diagnosis system based on multimodal fusion and knowledge graph, characterized in that, The system comprises: Interaction layer: Used to provide various database status queries and statistics, various database operation and maintenance operations, database fault detection and handling, and database knowledge Q&A functions; Application layer: includes dedicated task intelligence agents and general task intelligence agents. The dedicated task intelligence agents include SQL optimization modules, inspection and analysis modules, and fault diagnosis modules, which are used to execute corresponding intelligent operation and maintenance tasks. The general task intelligence agents include API intelligence agents, which are used to perform intent recognition, target API selection, API call sequence planning, and API call execution to achieve automated handling of operation and maintenance tasks. Model service layer: includes general model and special model. The general model is a large language model. The special model includes machine learning model, deep learning model, statistical algorithm and special small model, which are used to calculate and rank attention of multimodal data and output a set of highly concerned object features. Data Layer: Used to store real-time private domain knowledge and general private domain knowledge. The real-time private domain knowledge includes basic resource information, time-series monitoring data, and object attribute information. The general private domain knowledge includes vectorized documents and knowledge graphs, and supports vectorized retrieval, graph query, relational data query, and time-series data query functions.

Images