Fault diagnosis question and answer system based on multi-modal knowledge graph and large language model

Abstract

The application relates to a kind of fault diagnosis question and answer systems based on multi-modal knowledge graph and large language model provided by the application, comprising: time series data processing module, for carrying out noise reduction processing to time series data, extracting features to the time series data after noise reduction, using multiple feature data to represent time series data ontology;Multi-modal knowledge graph module, for constructing multi-modal knowledge graph, including text modal knowledge graph and time series data modal knowledge graph;Large model fine-tuning module, for fine-tuning large model based on different fault problems and corresponding solutions, to obtain question and answer large model using LoRA model;RAG question and answer module, for enhancing user questions based on multi-modal knowledge graph, inputting the enhanced sentence into the fine-tuned large model, and outputting the result.The application improves the speed of fault diagnosis, realizes fault question and answer for various data types, and solves the problem of lack of professional knowledge for large model.

Description

Technical Field

[0001] The present invention belongs to the field of fault diagnosis technology, and in particular relates to a fault diagnosis question-answering system based on a multimodal knowledge graph and a large language model. Background Art

[0002] Traditional fault diagnosis systems rely on expert systems, physical models, or data-driven algorithms. These systems achieve automated fault detection and diagnosis to a certain extent, but with the increasing complexity of industrial systems, existing technologies are constantly evolving, especially in the following aspects:

[0003] 1. Fault diagnosis based on expert system

[0004] Early fault diagnosis systems often relied on expert systems. Expert systems collect knowledge from domain experts and form a rule base to facilitate fault diagnosis. These systems typically rely on rule matching. When a device experiences an anomaly, the system proposes a solution based on the information in the rule base.

[0005] 2. Fault diagnosis based on physical models

[0006] Physical models build mathematical models of a device or system, using the system's operating parameters and input-output relationships to predict and diagnose faults. These methods rely on a deep understanding of the device's operating process, converting its physical characteristics into model parameters to determine if a fault exists.

[0007] 3. Data-driven fault diagnosis

[0008] With the advancement of sensor technology, fault diagnosis techniques based on sensor data have gradually developed. These methods typically use machine learning or deep learning algorithms to analyze historical data, train models, and use these models to predict faults. Commonly used data includes vibration signals, temperature, sound waves, and images. Powered by big data and cloud computing, these systems can process massive amounts of equipment data and perform real-time diagnosis.

[0009] 4. Application of Knowledge Graph in Fault Diagnosis

[0010] Knowledge graphs, a technology for organizing and representing knowledge in a graph-like format, are widely used in knowledge management and intelligent question-answering systems. In recent years, knowledge graphs have been gradually introduced into the field of fault diagnosis. By structuring information about equipment, faults, and solutions, knowledge graphs can provide diagnostic systems with richer background knowledge and provide a basis for reasoning about equipment failures.

[0011] 5. Application of Large Language Model (LLM) in Intelligent Question Answering

[0012] Large language models, such as GPT-3 and BERT, have made significant progress in natural language processing, especially in intelligent question answering and semantic understanding. These models are trained on vast amounts of text data and can understand user questions and provide reasonable answers. Compared with traditional diagnostic systems, systems based on large language models can handle more complex natural language problems.

[0013] The prior art has the following technical defects:

[0014] 1. Over-reliance on human decision-making, knowledge is difficult to accumulate, inherit and reuse

[0015] In the process of diagnosing equipment failures, decision-makers need to make decisions based on their theoretical knowledge, which is often limited by the knowledge accumulation and maintenance experience of decision-makers. Knowledge-driven fault diagnosis relies heavily on knowledge accumulation, but in reality, knowledge is often fragmented and scattered, making it difficult to accumulate, inherit and reuse knowledge.

[0016] 2. Single modality problem of knowledge graph

[0017] Most existing knowledge graphs only process text or structured data and cannot effectively integrate and process multi-modal data such as images, sounds, and vibration signals. This limitation results in insufficient reasoning and diagnostic capabilities when the system encounters multi-modal device data.

[0018] 3. Lack of adaptability of large language models in professional fields

[0019] Although large language models perform well in natural language processing, they lack the deep integration of professional knowledge and structured knowledge in professional fields, especially in fault diagnosis, making it difficult to cope with complex industrial scenarios. In addition, the processing capabilities of language models for non-text data are limited, affecting their application effectiveness in actual fault diagnosis.

[0020] 4. Real-time and explainability issues

[0021] As diagnostic algorithms become more complex, the demand for computing resources increases significantly, and existing technologies struggle to provide high-accuracy diagnosis while ensuring real-time response. In addition, many existing machine learning algorithms lack explainability, making it difficult for users to understand the basis of the system's diagnosis and affecting their trust. SUMMARY

[0022] The purpose of the present application is to provide a fault diagnosis question and answer system based on multi-modal knowledge graph and large language model to solve the above technical problems.

[0023] The present application provides a fault diagnosis question and answer system based on multi-modal knowledge graph and large language model, comprising:

[0024] The time series data processing module is used to perform noise reduction on the time series data, extract features from the noise-reduced time series data, and use multiple feature data to represent the time series data itself;

[0025] Multimodal knowledge graph module, used to build multimodal knowledge graphs, including text modal knowledge graphs and time series data modal knowledge graphs;

[0026] The large model fine-tuning module is used to fine-tune the large model based on different fault problems and their corresponding solutions using the LoRA model to obtain a large question-answering model;

[0027] The RAG question-answering module is used to enhance user questions based on the multimodal knowledge graph, input the enhanced sentences into the fine-tuned large model, and output the results.

[0028] Furthermore, the multimodal knowledge graph module uses a deep learning model to perform named entity recognition, performs entity extraction, relationship extraction, and attribute extraction on text modal knowledge, fuses the extracted triples, constructs a text modal knowledge graph, and saves the constructed text modal knowledge graph into a graph database.

[0029] Furthermore, the text modal knowledge includes emergency repair work orders, maintenance manuals, equipment failures, system logs and operation manuals.

[0030] Furthermore, the multimodal knowledge graph module uses the extracted feature data as child nodes to construct a time series data modal knowledge graph.

[0031] Furthermore, the large model fine-tuning module is specifically used to:

[0032] ① Add a bypass to the cardinality of the original pre-trained model to simulate the intrinsic rank by first reducing the dimension and then increasing the dimension;

[0033] ② Initialize the A and B matrices with random Gaussian distribution and zero matrix respectively. During training, fix the parameters of the pre-trained model and only train the parameters of matrix A and matrix B.

[0034] ③ After training is completed, matrix B is multiplied by matrix A and the pre-trained model parameters are merged as the fine-tuned large model parameters.

[0035] Furthermore, the RAG question-answering module is specifically used to:

[0036] Determine whether the content of the user's question is multimodal data. If it is a mixture of time series data and text data, use the time series data processing module to perform data noise reduction and feature extraction on the time series data, and use the named entity recognition module to extract text entities and identify intents on the text data;

[0037] After the data is processed, it is queried through the multimodal knowledge graph, the query results are spliced ​​into the original user question, the spliced ​​content is input into the fine-tuned large model, and answers are given based on the existing knowledge, and finally the answers are output.

[0038] The above solution, through a fault diagnosis question-answering system based on a multimodal knowledge graph and a large language model, achieves the following technical effects:

[0039] (1) It improves the speed of fault diagnosis, reduces the knowledge reserve of fault diagnosis personnel, and realizes simple and efficient updating of knowledge base. Maintenance personnel can complete the task of mechanical equipment fault diagnosis based on simple dialogue. In addition, non-professionals can supplement and modify the multimodal knowledge graph through the web page.

[0040] (2) Implemented fault Q&A for multiple data types. A multimodal knowledge graph was constructed, and through early data processing, a fault analysis and Q&A system for time series data and text data was implemented. This solved the current single-source Q&A problem of only answering questions for text data.

[0041] (3) By constructing a multimodal knowledge graph and fine-tuning the large model, the problem of the large model's lack of professional domain knowledge is solved. By performing entity recognition and feature extraction on the question and searching the multimodal knowledge graph, the question content is supplemented. The question with supplemented professional knowledge is input into the fine-tuned large model, and the question-answering result is finally output.

[0042] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of the structure of the fault diagnosis question-answering system based on a multimodal knowledge graph and a large language model of the present invention;

[0044] Figure 2 A flowchart for constructing a text modality knowledge graph in one embodiment of the present invention;

[0045] Figure 3 This is a flowchart for constructing a time series data modal knowledge graph in one embodiment of the present invention;

[0046] Figure 4 is a LoRA model in one embodiment of the present invention;

[0047] Figure 5 FIG. 4 is a flowchart of a RAG question-and-answer process in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The specific embodiments of the present application are described in further detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0049] Referring to the drawings, Figure 1 The present embodiment provides a fault diagnosis question and answer system based on a multi-modal knowledge graph and a large language model, characterized by comprising:

[0050] a time series data processing module, configured to perform noise reduction processing on time series data, perform feature extraction on the noise-reduced time series data, and use a plurality of feature data to represent the time series data entity;

[0051] a multi-modal knowledge graph module, configured to construct a multi-modal knowledge graph, including a text modal knowledge graph and a time series data modal knowledge graph;

[0052] a large model fine-tuning module, configured to fine-tune a large model using a LoRA model based on different fault problems and their corresponding solutions to obtain a question and answer large model;

[0053] a RAG question and answer module, configured to perform enhanced processing on a user question based on a multi-modal knowledge graph, input the enhanced sentence into the fine-tuned large model, and output a result.

[0054] The fault diagnosis question and answer system constructs a multi-modal knowledge graph. In a knowledge-driven fault diagnosis scheme, relying on a large number of maintenance logs and expert experience, triple information is extracted from maintenance records through natural language processing technology, and stored in a graph database to construct a knowledge graph. The knowledge graph not only contains text information, but also integrates multi-modal data such as vibration signals and their features, so as to comprehensively display multi-source knowledge in fault diagnosis. In addition, the system supports experts to supplement or revise knowledge items on the front-end page, and the submitted content will be directly updated to the graph database without the need for experts to master Cypher and other graph database query languages. In this way, the knowledge graph not only maintains efficient management and updating of data, but also ensures convenient integration of expert knowledge, enhancing the accuracy and practicality of the fault diagnosis system.

[0055] The fine-tuning of the large language model in the fault diagnosis question and answer system is based on the Low-Rank Adaptation (LoRA) fine-tuning method, which enables the large model to quickly adapt to new professional fields and tasks with minimal storage and computing overhead. This method freezes the original weights and decomposes the weight matrix that needs to be adjusted into two matrices A and B through low-rank matrix decomposition. The rank size can be adjusted according to the application. In the reasoning phase, the optimized results of the low-rank matrix are merged into the original model weights, thereby realizing the improvement of the domain adaptability of the model without affecting the general knowledge structure of the model.

[0056] The fault diagnosis question-answering system uses a multimodal knowledge graph as the knowledge base, and is designed with a fine-tuned large language model to perform a fault diagnosis question-answering system. For text-type input, named entity recognition and intent recognition are performed on the user's input questions. The entities identified by NER are used to query the knowledge graph, and the identified entities are matched with the nodes in the knowledge graph to find relevant information. The intent recognition results are used in combination with the entities and relationships of the knowledge graph to determine the relationship path between the user's question and the graph. According to the question type (such as cause and effect, definition, process, etc.), the corresponding path and node are found, and the content in the knowledge base is used as the preliminary answer. The preliminary answer or triple information of the knowledge graph query is passed to the large model, and it is used to generate a natural language answer. The large model can convert structured data (such as triples) into fluent answers to enhance the user experience. For example, if you input "the cause of the failure of device A", the large model can generate a complete sentence description based on the graph query results. For the input of vibration signals, the vibration signals are first analyzed, and the knowledge graph is queried for information such as its frequency range and amplitude peak. The relevant information of the fault type is retrieved, and the text knowledge graph is continued to be searched to find relevant information. The local information retrieved is input into the large model, and the fault phenomenon and solution are returned.

[0057] The present invention is described in further detail below.

[0058] 1. Time Series Data Processing Module

[0059] The time series data processing module uses signal processing technology to reduce noise and extract features from time series signals. First, the data is denoised, and then the main feature signals are extracted from the denoised data. Multiple feature signals are used to represent the time series data itself.

[0060] This module provides technical support for the construction of the time series data modal knowledge graph. It saves the denoised time series data as entity nodes in the knowledge graph, and uses its feature data as child nodes to construct the time series data modal knowledge graph.

[0061] In the RAG question-answering module, this module will also perform noise reduction feature extraction on the time series data passed in by the user, and then search for similar signals in the multimodal knowledge base, enabling the question-answering system to process and analyze time series data.

[0062] 2. Multimodal Knowledge Graph Module

[0063] (1) Construction of text modal knowledge graph

[0064] Text modal knowledge comes from the company's internal repair work orders, maintenance manuals, equipment failures, system logs and operation manuals. Its data is mainly structured data. Knowledge is extracted from semi-structured and unstructured data. A deep learning model is used for named entity recognition, entity extraction and relationship extraction. Finally, the extracted triples are fused to form a knowledge graph. The constructed knowledge graph is saved in the Neo4j graph database for persistence. Figure 2 shown.

[0065] (2) Construction of time series data modal knowledge graph

[0066] Time series data comes from signals obtained in advance from experiments or historical fault signals. First, the time series data is subjected to noise reduction and then feature extraction. The features extracted from the signal are used as the attributes of the signal. The signal is used as the entity node, and the signal features decomposed from the signal are used as the child nodes of the signal, thereby constructing a knowledge graph. Figure 3 shown.

[0067] 3. Large Model Fine-tuning Module

[0068] The large model fine-tuning module is used to fine-tune the large model based on the different fault problems and their corresponding solutions generated by the fault problem and solution generation module to obtain a large question-answering model. The LoRA model is used to fine-tune the large model. The principle diagram is as follows: Figure 4 The process is as follows:

[0069] ① Add a bypass to the cardinality of the original pre-trained model, first perform dimensionality reduction and then dimensionality increase operations to simulate the intrinsic rank.

[0070] ② The A and B matrices are initialized with random Gaussian distribution and zero matrix respectively. During training, the parameters of the pre-trained model are fixed, and only the parameters of matrix A and matrix B are trained.

[0071] ③ After training is completed, matrix B is multiplied by matrix A and the pre-trained model parameters are combined as the fine-tuned model parameters.

[0072] 4. RAG Question and Answer Module

[0073] The RAG (Retrieval-Augmented Generation) question-answering module integrates the above content to complete the retrieval-augmented question-answering task. Its process is as follows: Figure 5 The main task of this module is to enhance user questions according to the multimodal knowledge graph, input the enhanced sentences into the fine-tuned large model, and finally output the results.

[0074] First, we determine whether the user's question is multimodal data. If it is a mixture of time series and text data, the time series data is processed through the time series data processing module for noise reduction and feature extraction, and the text data is processed through the named entity recognition module for text entity extraction and intent recognition. After processing, the data is queried in the multimodal knowledge graph, and the query results are spliced ​​into the original user question. This spliced ​​content is then fed into the fine-tuned large model, which responds based on the existing knowledge and outputs the answer.

[0075] Through the above technical solution, the present invention solves the following technical problems:

[0076] 1) Solve the problem of over-reliance on manual decision-making and difficulty in accumulating, inheriting, and reusing knowledge:

[0077] In the process of fault diagnosis, decision makers need to make decisions based on their own experience. However, the differences in the amount of knowledge accumulated by decision makers often lead to certain limitations in the decision-making methods. The present invention uses a knowledge base, so that decision makers can use the historical knowledge in the knowledge base to assist in fault diagnosis during the fault diagnosis process, thereby improving the accuracy of diagnosis. In the actual equipment fault diagnosis process, it is necessary to rely on a large amount of expert knowledge, maintenance manuals, equipment instructions and other knowledge. The knowledge is fragmented and scattered, making it difficult to accumulate, pass on and reuse. The present invention uses a graph database to build a question-and-answer system, saves production knowledge in the graph database, and solves the problem that knowledge is difficult to accumulate, pass on and reuse.

[0078] 2) Solve the single modality problem of knowledge graph:

[0079] By incorporating multimodal data from text and vibration signals, a knowledge graph is constructed. Within the knowledge graph, data from different modalities is represented as nodes. For the text modality, nodes typically represent entities such as equipment, fault type, cause, and solution. For the vibration signal modality, nodes represent different signal characteristics, such as frequency range and peak amplitude. By converting these different modal data into nodes within the graph, the system can unify information from different data sources and provide a foundation for subsequent reasoning.

[0080] 3) This solves the problem of insufficient adaptability of large language models in professional fields:

[0081] By fine-tuning the large model and the multimodal knowledge graph as RAG, the problem of insufficient adaptability of the language model in professional fields is solved. By training the faults and their solutions, the large model is given a certain level of professional knowledge, while the multimodal knowledge graph has the ability to analyze time series data. The multimodal knowledge graph integrates multi-source heterogeneous data such as text and vibration signals, and stores this multimodal information in a structured form in the graph. During application, the system can not only directly provide structured information such as background knowledge and domain terminology in the professional field to the large language model, but also improve the language model's understanding and response capabilities to professional problems through the semantic fusion of multimodal data. Through this knowledge enhancement method, the large language model can more accurately identify and parse professional terms and complex concepts, thereby providing more accurate and applicable answers in diagnostic questions and technical consultations, significantly improving the model's adaptability and practicality in professional scenarios.

[0082] 4) Solved the real-time and explainability issues:

[0083] Knowledge graphs are highly structured and organized, enabling real-time fault diagnosis through rapid retrieval and reasoning. Compared to traditional machine learning algorithms, which require extensive computing resources and training time, multimodal knowledge graphs, by augmenting prior knowledge, can effectively reduce the amount of computation required during the diagnostic process and improve the system's real-time response capabilities. Knowledge graphs themselves are highly interpretable, with all nodes and edges representing clear relationships between entities such as equipment, faults, and solutions. Combined with a large language model, the system can provide natural language explanations to help users understand the root cause of faults and solutions. This knowledge-based reasoning process is easier to understand and trust than traditional "black box" machine learning models, addressing the lack of interpretability in existing technologies.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A fault diagnosis question-answering system based on multimodal knowledge graph and large language model, characterized by: include: The time series data processing module is used to perform noise reduction on the time series data, extract features from the noise-reduced time series data, and use multiple feature data to represent the time series data itself; Multimodal knowledge graph module, used to build multimodal knowledge graphs, including text modal knowledge graphs and time series data modal knowledge graphs; The large model fine-tuning module is used to fine-tune the large model based on different fault problems and their corresponding solutions using the LoRA model to obtain a large question-answering model, including: ① Add a bypass to the cardinality of the original pre-trained model to simulate the intrinsic rank by first reducing the dimension and then increasing the dimension; ② Initialize the A and B matrices with random Gaussian distribution and zero matrix respectively. During training, fix the parameters of the pre-trained model and only train the parameters of matrix A and matrix B. ③After training is completed, matrix B is multiplied by matrix A and the pre-trained model parameters are combined as the fine-tuned large model parameters; The RAG question-answering module is used to enhance user questions based on the multimodal knowledge graph, input the enhanced sentences into the fine-tuned large model, and output the results, including: Determine whether the content of the user's question is multimodal data. If it is a mixture of time series data and text data, use the time series data processing module to perform data noise reduction and feature extraction on the time series data, and use the named entity recognition module to extract text entities and identify intents on the text data; After the data is processed, it is queried through the multimodal knowledge graph, the query results are spliced ​​into the original user question, the spliced ​​content is input into the fine-tuned large model, and answers are given based on the existing knowledge, and finally the answers are output.

2. The fault diagnosis question-answering system based on multimodal knowledge graph and large language model according to claim 1 is characterized in that: The multimodal knowledge graph module uses a deep learning model to perform named entity recognition, extract entities, extract relationships, and extract attributes from text modal knowledge, fuse the extracted triples, construct a text modal knowledge graph, and save the constructed text modal knowledge graph into a graph database.

3. The fault diagnosis question-answering system based on multimodal knowledge graph and large language model according to claim 2, characterized in that: The text modal knowledge includes emergency repair work orders, maintenance manuals, equipment failures, system logs and operation manuals.

4. The fault diagnosis question-answering system based on multimodal knowledge graph and large language model according to claim 3 is characterized in that: The multimodal knowledge graph module uses the extracted feature data as child nodes to construct a time series data modal knowledge graph.

Images