Fault diagnosis question-answering system based on multi-modal knowledge graph and large language model

Abstract

The invention provides a fault diagnosis question-answering system based on a multi-modal knowledge graph and a large language model, and the system comprises a time sequence data processing module which is used for carrying out the noise reduction of time sequence data, carrying out the feature extraction of the time sequence data after noise reduction, and representing a time sequence data body through a plurality of feature data; the multi-modal knowledge graph module is used for constructing a multi-modal knowledge graph, and the multi-modal knowledge graph comprises a text modal knowledge graph and a time sequence data modal knowledge graph; the large model fine tuning module is used for performing fine tuning on the large model by adopting a LoRA model based on different fault problems and corresponding solutions so as to obtain a question and answer large model; and the RAG question and answer module is used for enhancing the user question based on the multi-modal knowledge graph, inputting the enhanced statement into the fine-tuned large model, and outputting a result. According to the method, the fault diagnosis speed is increased, fault questions and answers are carried out on various data types, and the problem that a large model lacks professional domain knowledge is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of fault diagnosis, and in particular relates to a fault diagnosis question-answering system based on a multimodal knowledge graph and a large language model. Background Art

[0002] Traditional fault diagnosis systems rely on expert systems, physical models, or data-driven algorithms. These systems have achieved automated fault detection and diagnosis to a certain extent, but with the increasing complexity of industrial systems, existing technologies are also evolving, especially in the following aspects:

[0003] 1. Fault diagnosis based on expert system

[0004] Early fault diagnosis systems usually rely on expert systems. Expert systems collect knowledge from domain experts to form a rule base to achieve fault diagnosis. Such systems are usually based on rule matching. When a device fails, the system will propose a solution based on the information in the rule base.

[0005] 2. Fault diagnosis based on physical models

[0006] The physical model is to predict and diagnose faults by building a mathematical model of the device or system based on the system's operating parameters and input-output relationship. This type of method relies on a deep understanding of the device's operating process, converting the device's physical characteristics into model parameters to determine whether there is a fault.

[0007] 3. Data-driven fault diagnosis

[0008] With the development of sensor technology, fault diagnosis technology based on sensor data has gradually developed. This type of method usually uses machine learning or deep learning algorithms to analyze historical data, train models, and predict the occurrence of faults through models. Commonly used data includes vibration signals, temperature, sound waves, images, etc. With the support of big data and cloud computing, this type of system can process massive equipment data and perform real-time diagnosis.

[0009] 4. Application of knowledge graph in fault diagnosis

[0010] Knowledge graph is a technology that organizes and expresses knowledge in the form of a graph structure. It is widely used in knowledge management and intelligent question-answering systems. In recent years, knowledge graphs have been gradually introduced into the field of fault diagnosis. By structuring information such as equipment, faults, and solutions, knowledge graphs can provide richer background knowledge for diagnostic systems and provide a basis for reasoning about equipment failures.

[0011] 5. Application of Large Language Model (LLM) in Intelligent Question Answering

[0012] Large language models, such as GPT-3 and BERT, have made significant progress in the field of natural language processing, especially in intelligent question answering and semantic understanding. These models are trained with massive amounts of text data and can understand user questions and provide reasonable answers. Compared with traditional diagnostic systems, systems based on large language models can handle more complex natural language problems.

[0013] The prior art has the following technical defects:

[0014] 1. Over-reliance on manual decision-making makes it difficult to accumulate, inherit and reuse knowledge

[0015] In the process of equipment fault diagnosis, decision makers need to make decisions based on their own theoretical knowledge, which is often limited by their knowledge accumulation and maintenance experience. Knowledge-driven fault diagnosis relies heavily on knowledge accumulation, but in reality, knowledge is often fragmented and dispersed, making it impossible to accumulate, inherit, and reuse knowledge.

[0016] 2. Single modality problem of knowledge graph

[0017] Most existing knowledge graphs only process text or structured data and cannot effectively integrate and process multimodal data (such as images, sounds, vibration signals, etc.). This limitation results in insufficient reasoning and diagnostic capabilities of the system when faced with multimodal device data.

[0018] 3. Large language models are not adaptable enough in professional fields

[0019] Although large language models have excellent performance in the field of natural language processing, they lack the deep combination of professional knowledge and structured knowledge in professional fields, especially fault diagnosis, and are difficult to cope with complex industrial scenarios. In addition, language models have limited processing capabilities for non-text data, which affects their application effect in actual fault diagnosis.

[0020] 4. Real-time and explainability issues

[0021] As diagnostic algorithms become more complex, the system's computing resource requirements have increased significantly, and existing technologies have difficulty providing high-accuracy diagnoses while ensuring real-time responses. In addition, many existing machine learning algorithms lack interpretability, making it difficult for users to understand the system's diagnostic basis, affecting their trust. Summary of the invention

[0022] The purpose of the present invention is to provide a fault diagnosis question-answering system based on a multimodal knowledge graph and a large language model to solve the above-mentioned technical problems.

[0023] The present invention provides a fault diagnosis question-answering system based on a multimodal knowledge graph and a large language model, comprising:

[0024] A time series data processing module is used to perform noise reduction on the time series data, extract features from the noise-reduced time series data, and use multiple feature data to represent the time series data body;

[0025] Multimodal knowledge graph module, used to build multimodal knowledge graphs, including text modal knowledge graphs and time series data modal knowledge graphs;

[0026] The large model fine-tuning module is used to fine-tune the large model using the LoRA model based on different fault problems and their corresponding solutions to obtain a large question-answering model;

[0027] The RAG question-answering module is used to enhance user questions based on the multimodal knowledge graph, input the enhanced sentences into the fine-tuned large model, and output the results.

[0028] Furthermore, the multimodal knowledge graph module uses a deep learning model to perform named entity recognition, performs entity extraction, relationship extraction, and attribute extraction on text modal knowledge, fuses the extracted triples, constructs a text modal knowledge graph, and saves the constructed text modal knowledge graph into a graph database.

[0029] Furthermore, the text modal knowledge includes emergency repair work orders, maintenance manuals, equipment failures, system logs and operation manuals.

[0030] Furthermore, the multimodal knowledge graph module uses the extracted feature data as child nodes to construct a time series data modal knowledge graph.

[0031] Furthermore, the large model fine-tuning module is specifically used for:

[0032] ① Add a bypass to the cardinality of the original pre-trained model to simulate the intrinsic rank by first reducing the dimension and then increasing the dimension;

[0033] ② Initialize the A and B matrices with random Gaussian distribution and zero matrix respectively. During training, fix the parameters of the pre-trained model and only train the parameters of matrix A and matrix B.

[0034] ③After training is completed, matrix B is multiplied by matrix A and the pre-trained model parameters are combined as the fine-tuned large model parameters.

[0035] Furthermore, the RAG question-answering module is specifically used for:

[0036] Determine whether the content of the user's question is multimodal data. If it is a mixture of time series data and text data, perform data noise reduction and feature extraction on the time series data through the time series data processing module, and perform text entity extraction and intent recognition on the text data through the named entity recognition module;

[0037] After the data is processed, it is queried through the multimodal knowledge graph, the query results are spliced ​​into the original user questions, the spliced ​​content is input into the fine-tuned large model, answers are given based on existing knowledge, and finally the answers are output.

[0038] Through the above solution, the fault diagnosis question-answering system based on multimodal knowledge graph and large language model has the following technical effects:

[0039] (1) It improves the speed of fault diagnosis, reduces the knowledge reserve of fault diagnosis personnel, and realizes simple and efficient updating of knowledge base. Maintenance personnel can complete the task of mechanical equipment fault diagnosis based on simple dialogue. In addition, non-professionals can supplement and modify the multimodal knowledge graph through the web page.

[0040] (2) Fault Q&A for multiple data types is realized. A multimodal knowledge graph is constructed, and through preliminary data processing, a fault analysis and Q&A system for time series data and text data is realized. This solves the problem of single Q&A that only answers text data.

[0041] (3) By constructing a multimodal knowledge graph and fine-tuning the large model, the problem of the large model's lack of professional knowledge is solved. By performing entity recognition and feature extraction on the question and searching the multimodal knowledge graph, the question content is supplemented, and the question with supplemented professional knowledge is input into the fine-tuned large model, and finally the question and answer results are output.

[0042] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural schematic diagram of a fault diagnosis question-answering system based on a multimodal knowledge graph and a large language model of the present invention;

[0044] Figure 2 A flowchart for constructing a text modality knowledge graph in an embodiment of the present invention;

[0045] Figure 3 A flowchart for constructing a temporal data modal knowledge graph in one embodiment of the present invention;

[0046] Figure 4 is a LoRA model in one embodiment of the present invention;

[0047] Figure 5 The figure is a RAG question-and-answer flow chart in one embodiment of the present invention. DETAILED DESCRIPTION

[0048] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0049] Ginseng Figure 1 As shown, this embodiment provides a fault diagnosis question-answering system based on a multimodal knowledge graph and a large language model, which is characterized by including:

[0050] A time series data processing module is used to perform noise reduction on the time series data, extract features from the noise-reduced time series data, and use multiple feature data to represent the time series data body;

[0051] Multimodal knowledge graph module, used to build multimodal knowledge graphs, including text modal knowledge graphs and time series data modal knowledge graphs;

[0052] The large model fine-tuning module is used to fine-tune the large model using the LoRA model based on different fault problems and their corresponding solutions to obtain a large question-answering model;

[0053] The RAG question-answering module is used to enhance user questions based on the multimodal knowledge graph, input the enhanced sentences into the fine-tuned large model, and output the results.

[0054] The fault diagnosis question-and-answer system has built a multimodal knowledge graph. In the knowledge-driven fault diagnosis solution, it relies on a large number of maintenance logs and expert experience, extracts triple information from maintenance records through natural language processing technology, and stores it in the graph database to build a knowledge graph. The knowledge graph not only contains text information, but also integrates multimodal data such as vibration signals and their characteristics, so as to fully display multi-source knowledge in fault diagnosis. In addition, the system supports experts to supplement or revise knowledge entries on the front-end page, and the submitted content will be directly updated to the graph database without the need for experts to master graph database query languages ​​such as Cypher. In this way, the knowledge graph not only maintains the efficient management and updating of data, but also ensures the convenient integration of expert knowledge, enhancing the accuracy and practicality of the fault diagnosis system.

[0055] In this fault diagnosis question-answering system, the fine-tuning of the large language model is based on the fine-tuning method of Low-Rank Adaptation (LoRA), which enables the large model to quickly adapt to new professional fields and tasks with minimal storage and computing overhead. This method freezes the original weights and decomposes the part of the weight matrix that needs to be adjusted into two matrices, A and B, through a low-rank matrix. The rank size can be adjusted according to the application. In the inference stage, the optimized result of the low-rank matrix is ​​merged into the original model weight, thereby improving the domain adaptability of the model without affecting the general knowledge structure of the model.

[0056] The fault diagnosis question-and-answer system, the question-and-answer architecture design uses a multimodal knowledge graph as the knowledge base, and cooperates with a fine-tuned large language model to make a fault diagnosis question-and-answer system. For text-type input, the user's input question is subjected to named entity recognition and intent recognition. The entities identified by NER are used to query the knowledge graph, and the identified entities are matched with the nodes in the knowledge graph to find relevant information. The intent recognition results are used in combination with the entities and relationships of the knowledge graph to determine the relationship path between the user's question and the graph. According to the type of question (such as cause and effect, definition, process, etc.), the corresponding path and node are found, and the content in the knowledge base is used as a preliminary answer. The preliminary answer or triple information of the knowledge graph query is passed to the large model, and it is used to generate a natural language answer. The large model can convert structured data (such as triples) into fluent answers to enhance the user experience. For example, if you input "the cause of the failure of device A", the large model can generate a complete sentence description based on the graph query results. For the input of vibration signal, the vibration signal is first analyzed, and the knowledge graph is queried for its frequency range, amplitude peak and other information. The relevant information of the fault type is retrieved, and the text knowledge graph is continued to be searched to find relevant information. The queried local information is input into the big model, and the fault phenomenon and solution are returned.

[0057] The present invention is described in further detail below.

[0058] 1. Time Series Data Processing Module

[0059] The time series data processing module uses signal processing technology to perform noise reduction and feature extraction on time series signals. First, the data is subjected to noise reduction, and then the main feature signals are extracted from the noise-reduced data. Multiple feature signals are used to represent the time series data itself.

[0060] This module provides technical support for the construction of the time series data modal knowledge graph. It saves the denoised time series data as entity nodes in the knowledge graph, and uses its feature data as child nodes to construct the time series data modal knowledge graph.

[0061] In the RAG question-answering module, this module will also perform noise reduction feature extraction on the time series data passed in by the user, and then search for similar signals in the multimodal knowledge base, so that the question-answering system has the ability to process and analyze time series data.

[0062] 2. Multimodal Knowledge Graph Module

[0063] (1) Construction of text modal knowledge graph

[0064] Text modal knowledge comes from the repair work orders, maintenance manuals, equipment failures, system logs and operation manuals within the enterprise. The data is mainly structured data. Knowledge is extracted from semi-structured and unstructured data. A deep learning model is used for named entity recognition, entity extraction and relationship extraction. Finally, the extracted triples are fused to construct a knowledge graph. The constructed knowledge graph is saved in the Neo4j graph database for persistence. Figure 2 shown.

[0065] (2) Construction of time series data modal knowledge graph

[0066] The time series data comes from the signals obtained in advance from experiments or historical fault signals. First, the time series data is subjected to noise reduction and then feature extraction. The features extracted from the signal are used as the attributes of the signal. The signal is used as an entity node, and the signal features decomposed from the signal are used as the child nodes of the signal, thereby constructing a knowledge graph. Figure 3 shown.

[0067] 3. Large Model Fine-tuning Module

[0068] The large model fine-tuning module is used to fine-tune the large model based on the different fault problems and their corresponding solutions generated by the fault problem and solution generation module to obtain a large question-answering model. The LoRA model is used to fine-tune the large model. The schematic diagram is as follows: Figure 4 The process is as follows:

[0069] ① Add a bypass to the base of the original pre-trained model, first perform dimensionality reduction and then dimensionality increase operations to simulate the intrinsic rank.

[0070] ② The A and B matrices are initialized with random Gaussian distribution and zero matrix respectively. During training, the parameters of the pre-trained model are fixed, and only the parameters of matrix A and matrix B are trained.

[0071] ③After training is completed, matrix B is multiplied by matrix A and the pre-trained model parameters are combined as the fine-tuned model parameters.

[0072] 4. RAG Question and Answer Module

[0073] The RAG (Retrieval-Augmented Generation) question-answering module integrates the above content to complete the retrieval-augmented generation question-answering task. Its process is as follows: Figure 5 The main task of this module is to enhance the user questions according to the multimodal knowledge graph, input the enhanced sentences into the fine-tuned large model, and finally output the results.

[0074] First, determine whether the content of the user's question is multimodal data. If it is a mixture of time series data and text data, then the time series data must be processed by the time series data processing module to reduce noise and extract features; the text data must be processed by the named entity recognition module to extract text entities and identify intent. After the data is processed, it is queried in the multimodal knowledge graph, and the query results are spliced ​​into the original user question. The spliced ​​content is then input into the fine-tuned large model, which answers based on the existing knowledge and finally outputs the answer.

[0075] Through the above technical solution, the present invention solves the following technical problems:

[0076] 1) Solved the problem of over-reliance on manual decision-making and difficulty in accumulating, inheriting and reusing knowledge:

[0077] In the process of fault diagnosis, decision makers need to make decisions based on their own experience, but the difference in the amount of knowledge accumulated by decision makers often leads to certain limitations in the decision-making methods. In the present invention, by using a knowledge base, decision makers can use the historical knowledge in the knowledge base to assist in fault diagnosis during fault diagnosis, thereby improving the accuracy of diagnosis. In the actual equipment fault diagnosis process, it is necessary to rely on a large amount of expert knowledge, maintenance manuals, equipment instructions and other knowledge, which is fragmented and scattered, and difficult to accumulate, inherit and reuse. The present invention uses a graph database to build a question-and-answer system, saves production knowledge in the graph database, and solves the problem that knowledge is difficult to accumulate, inherit and reuse.

[0078] 2) Solve the single modality problem of knowledge graph:

[0079] By introducing text and vibration signal multimodal data to build a knowledge graph, data of different modes are represented in the form of nodes in the knowledge graph. For text mode, nodes usually represent entities such as equipment, fault type, cause, solution, etc.; for vibration signal mode, nodes represent different signal characteristics, such as frequency range, amplitude peak, etc. By converting these different modal data into nodes in the graph, the system can unify the information from different data sources and provide a basis for subsequent reasoning.

[0080] 3) Solve the problem of insufficient adaptability of large language models in professional fields:

[0081] By fine-tuning the large model and the multimodal knowledge graph as RAG, the problem of insufficient adaptability of the language model in professional fields is solved. By training the faults and their solutions, the large model has certain professional knowledge, and the multimodal knowledge graph has the ability to analyze time series data. The multimodal knowledge graph integrates multi-source heterogeneous data such as text and vibration signals, and stores these multimodal information in a structured form in the graph. During the application process, the system can not only directly supply structured information such as background knowledge and domain terminology in the professional field to the large language model, but also improve the language model's understanding and response capabilities to professional issues through the semantic fusion of multimodal data. Through this knowledge enhancement method, the large language model can more accurately identify and parse professional terms and complex concepts, thereby providing more accurate and more applicable answers in diagnostic questions and technical consultations, significantly improving the adaptability and practicality of the model in professional scenarios.

[0082] 4) Solved the real-time and explainability issues:

[0083] Knowledge graphs are highly structured and organized, and can provide real-time fault diagnosis through rapid retrieval and reasoning. Compared with traditional machine learning algorithms that require a lot of computing resources and training time, multimodal knowledge graphs can effectively reduce the amount of computing in the diagnosis process and improve the real-time response capability of the system by enhancing prior knowledge. The knowledge graph itself has strong interpretability, and all nodes and edges represent clear relationships between entities such as equipment, faults, and solutions. Combined with a large language model, the system can provide natural language explanations to help users understand the root causes and solutions of faults. This knowledge-based reasoning process is easier to understand and trust than traditional "black box" machine learning models, solving the problem of insufficient interpretability in existing technologies.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the technical principles of the present invention, and these improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A fault diagnosis question-answering system based on multimodal knowledge graph and large language model, characterized in that: include: A time series data processing module is used to perform noise reduction on the time series data, extract features from the noise-reduced time series data, and use multiple feature data to represent the time series data body; Multimodal knowledge graph module, used to build multimodal knowledge graphs, including text modal knowledge graphs and time series data modal knowledge graphs; The large model fine-tuning module is used to fine-tune the large model using the LoRA model based on different fault problems and their corresponding solutions to obtain a large question-answering model; The RAG question-answering module is used to enhance user questions based on the multimodal knowledge graph, input the enhanced sentences into the fine-tuned large model, and output the results.

2. The fault diagnosis question-answering system based on multimodal knowledge graph and large language model according to claim 1, characterized in that: The multimodal knowledge graph module uses a deep learning model to perform named entity recognition, performs entity extraction, relationship extraction, and attribute extraction on text modal knowledge, fuses the extracted triples, constructs a text modal knowledge graph, and saves the constructed text modal knowledge graph into a graph database.

3. The fault diagnosis question-answering system based on multimodal knowledge graph and large language model according to claim 2, characterized in that: The text modal knowledge includes emergency repair work orders, maintenance manuals, equipment failures, system logs and operation manuals.

4. The fault diagnosis question-answering system based on multimodal knowledge graph and large language model according to claim 3, characterized in that: The multimodal knowledge graph module uses the extracted feature data as child nodes to construct a time series data modal knowledge graph.

5. The fault diagnosis question-answering system based on multimodal knowledge graph and large language model according to claim 4, characterized in that: The large model fine-tuning module is specifically used for: ① Add a bypass to the cardinality of the original pre-trained model to simulate the intrinsic rank by first reducing the dimension and then increasing the dimension; ② Initialize the A and B matrices with random Gaussian distribution and zero matrix respectively. During training, fix the parameters of the pre-trained model and only train the parameters of matrix A and matrix B. ③After training is completed, matrix B is multiplied by matrix A and the pre-trained model parameters are combined as the fine-tuned large model parameters.

6. The fault diagnosis question-answering system based on multimodal knowledge graph and large language model according to claim 5, characterized in that: The RAG question-answering module is specifically used for: Determine whether the content of the user's question is multimodal data. If it is a mixture of time series data and text data, perform data noise reduction and feature extraction on the time series data through the time series data processing module, and perform text entity extraction and intent recognition on the text data through the named entity recognition module; After the data is processed, it is queried through the multimodal knowledge graph, the query results are spliced ​​into the original user questions, the spliced ​​content is input into the fine-tuned large model, answers are given based on existing knowledge, and finally the answers are output.

Images