Power generation equipment multi-modal knowledge question and answer method based on retrieval enhancement generation

Abstract

The invention discloses a power generation equipment multi-mode question and answer method based on a retrieval enhancement generation technology. The method comprises the following steps: S1, collecting image-text data; s2, data cleaning and labeling; s3, performing fine tuning on the multi-modal large model based on an LoRA algorithm, and obtaining the multi-modal large model suitable for a power generation equipment question and answer scene by observing the test data set; s4, constructing a multi-modal equipment maintenance knowledge question-answering system; and S5, performing retrieval and generation based on the user question, extracting related text and image information from the knowledge base through a vector retrieval technology by the system according to the user question, and combining context cues to input the multi-modal large model to generate answer content containing answers and reference knowledge indexes. Through multi-modal data processing, a retrieval enhancement generation technology, reference traceability and dynamic updating and effective utilization of the knowledge base, the performance and practicability of the question and answer system in the field of power generation equipment management and maintenance are remarkably improved, and the comprehensive service capacity of the system is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of power generation industry equipment management and maintenance, and in particular to a multimodal knowledge question-answering method for power generation equipment based on retrieval enhancement generation. Background Art

[0002] The following major defects exist in the processing and application of multimodal data in the existing equipment maintenance field: 1. Single data mode processing: Existing systems can often only process a single type of data, such as text or signal data, and cannot perform comprehensive analysis on multiple types of data at the same time. This limits the scope of application and effectiveness of the system in complex industrial equipment maintenance scenarios. 2. Insufficient data integration and correlation: Even if some systems are able to process multimodal data, their ability to integrate multimodal data is relatively limited, and it is difficult to effectively associate information in different data modes, resulting in the inability to fully utilize the advantages of multimodal data. 3. Difficulties in updating and maintaining the knowledge base: The knowledge base update mechanism of the existing system is relatively lagging, and it is difficult to reflect the latest equipment information and maintenance standards in a timely manner, resulting in the possibility that the suggestions provided by the system may be outdated or inaccurate. Summary of the invention

[0003] In order to solve the existing problems, the present invention provides a multimodal knowledge question-answering method for power generation equipment based on retrieval enhancement generation. The specific scheme is as follows:

[0004] A multimodal question-answering method for power generation equipment based on retrieval enhancement generation technology comprises the following steps:

[0005] S1, collects graphic and text data related to power plant equipment;

[0006] S2, data cleaning and labeling;

[0007] S3, based on the LoRA algorithm and the data set processed in step S2, fine-tune the multimodal large model, and obtain a multimodal large model suitable for the power generation equipment question-answering scenario by observing the test data set;

[0008] S4, based on the pre-trained multimodal large model, a multimodal equipment maintenance knowledge question-answering system is constructed by retrieval enhancement to generate RAG;

[0009] S5, retrieval and generation based on user questions. The above system extracts relevant text and image information from the knowledge base through vector retrieval technology based on user questions, combines context prompt words into a multimodal large model, and generates answer content containing answers and reference knowledge indexes.

[0010] Preferably, the graphic data in step S1 include equipment operation and maintenance procedures, equipment factory report, technical test report, maintenance report, research report, and on-site equipment images.

[0011] Preferably, step S2 specifically includes: removing noise, duplicate data and irrelevant information through data cleaning, and producing a fine-tuning dataset in the form of image-text pairs through manual annotation, wherein the fine-tuning dataset is in the form of one image corresponding to one multi-round conversation record, providing high-quality training data for subsequent model fine-tuning.

[0012] Preferably, the specific steps of step S4 of constructing a multi-modal equipment maintenance knowledge question-answering system include:

[0013] S41, the system receives an input document;

[0014] S42, according to the file type of the document, the system selects a corresponding parser to convert the document content into two main forms: text and image;

[0015] S43, further processing the parsed text content and converting it into a vector representation;

[0016] S44, writing the storage address of the image on the server into the metadata portion of the corresponding vector to ensure the association between the image and the text vector;

[0017] S45, storing the generated vector data and image address information in the equipment maintenance knowledge base for subsequent retrieval and use;

[0018] S46, based on the pre-trained multimodal large model, a multimodal equipment maintenance knowledge question-answering system is constructed by retrieval enhancement to generate RAG.

[0019] Preferably, the document formats in step S41 include txt, markdown, docx and pdf; and the corresponding parsers in step S42 include xt file parser, markdown file parser, docx file parser and pdf file parser.

[0020] Preferably, step S5 specifically includes the following steps:

[0021] S51, starting from the user asking a question, extracting the text content in the user's question; inputting the extracted text into the embedding model, and retrieving the text, table and image address related to the user's question from the vector knowledge base through the text vector;

[0022] S52, filling the retrieved text and table into the context prompt word template to obtain the filled context prompt words;

[0023] S53, integrating the context prompt words and the image and text in the user's question to form a final input prompt word, which is input into a pre-trained multimodal large model to generate answer content;

[0024] S54, controlling the multimodal large model to generate content including answers and referenced knowledge index numbers through prompt words;

[0025] S55, outputting the generated answer and the referenced text knowledge index and the included images;

[0026] S56, determining whether the user needs to continue the conversation, if so, continuing the conversation, if not, ending the process.

[0027] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. After the computer program is run, any of the above methods is executed.

[0028] The present invention also discloses a computer system, including a processor and a storage medium, wherein a computer program is stored on the storage medium, and the processor reads and runs the computer program from the storage medium to execute any of the methods described above.

[0029] The beneficial effects of the present invention are:

[0030] The present invention significantly improves the processing capability of multimodal data. By constructing a fine-tuned dataset containing image and text information, the model can effectively understand and process complex problems, especially in scenarios where text and images need to be referenced at the same time. The retrieval-augmented generation (RAG) technology is used to retrieve relevant information from the vector knowledge base and generate answers, which solves the problem of the lack of external knowledge support in the traditional question-answering system and makes the answers more accurate and reliable. When generating answers, the referenced text knowledge index and image information are output at the same time, which enhances the intuitiveness and reference of the answers. The vector knowledge base contains a database of power plant equipment maintenance manuals, operating procedures, technical specifications and fault case databases. It is fine-tuned and aligned through the reinforcement learning method (RLHF) with human feedback and the LoRA algorithm to ensure that the model can answer questions based on the latest knowledge and standards, thereby improving the utilization efficiency and dynamic update capability of the knowledge base. The dialogue management module can determine whether the user needs to continue the dialogue, flexibly adjust the question-answering process, and improve the user experience. In general, the present invention significantly improves the performance and practicality of the question-and-answer system in the field of power generation equipment management and maintenance, and enhances the comprehensive service capabilities of the system through multimodal data processing, retrieval enhancement generation technology, reference traceability, and dynamic updating and effective use of the knowledge base. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 This is a flowchart for multimodal large model training;

[0033] Figure 2 It is a schematic diagram of a process for building an equipment maintenance knowledge base provided by an embodiment of the present invention;

[0034] Figure 3 It is a flowchart of intelligent question answering using a trained multimodal large model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] like Figure 1 It is a flowchart of multimodal large model training, showing the implementation steps of the method described in the present invention.

[0037] Specifically, a multimodal question-answering method for power generation equipment based on retrieval enhancement generation technology includes the following steps:

[0038] S1, collects graphic and text data related to power plant equipment.

[0039] Collect graphic data related to power plant equipment. The data includes but is not limited to equipment operation and maintenance procedures, equipment factory reports, technical test reports, maintenance reports, research reports and on-site equipment images. These data provide basic multimodal information for subsequent model training.

[0040] S2, perform data cleaning and labeling.

[0041] Data cleaning: Clean the collected graphic and text data. Specifically, remove noise, duplicate data, and irrelevant information in the data to ensure the quality and effectiveness of the data used for training.

[0042] Data annotation, i.e., the creation of a data set: Through manual annotation, the cleaned image and text data are created into an SFT (Supervised Fine-Tuning) data set in the form of image and text pairs. The SFT data set contains images and the corresponding multi-round conversation records, forming a one-to-one image and text pair data set for supervised fine-tuning of the model.

[0043] S3, model fine-tuning and model testing optimization.

[0044] Model fine-tuning: Based on the LoRA algorithm and the data set processed in step S2, the multimodal large model is fine-tuned to make it more suitable for question-answering scenarios related to power generation equipment. Through fine-tuning, the model can more effectively understand and process multimodal information in specific fields.

[0045] Model testing and optimization: By observing the performance of the test data set, optimizing and adjusting the model parameters, we finally obtained a multimodal large model suitable for power generation equipment question-answering scenarios. The model has undergone multiple rounds of testing and optimization to ensure that it can efficiently and accurately answer complex questions related to power generation equipment.

[0046] S4, based on the pre-trained multimodal large model, a multimodal equipment maintenance knowledge question-answering system is constructed by retrieval enhancement to generate RAG;

[0047] like Figure 2 , provides a flow chart of building a knowledge base for equipment maintenance, the specific steps include:

[0048] S41, document input: the system receives input documents. Document formats include but are not limited to txt, markdown, docx and pdf.

[0049] S42, Document Parsing and Content Conversion:

[0050] Document parsing: The system selects the corresponding parser based on the file type of the document. Parsers include but are not limited to txt file parsers, markdown file parsers, docx file parsers, pdf file parsers, etc.

[0051] Content conversion: After the selected parser parses the document, it converts the document content into two main forms: text and image. Text content includes text paragraphs, tables, image titles or descriptions, etc.

[0052] S43, further processing the parsed text content and converting it into a vector representation.

[0053] Specifically, (1) the parsed text content is further processed, especially the tabular data, image titles or descriptions, etc., to ensure accuracy in subsequent steps. (2) The processed text content is divided into blocks. The purpose of block processing is to split the long text content into smaller units for subsequent analysis and vectorization. (3) Use an embedding model (such as the BGE model) to vectorize the block text and convert the text content into a vector representation for fast retrieval in the knowledge base.

[0054] S44, image storage and image-text association:

[0055] Image storage: The parsed images are stored on the server and a unique storage address is generated for each image.

[0056] Image-text association: Write the storage address of the image on the server into the metadata part of the corresponding vector to ensure the association between the image and text vector.

[0057] S45, knowledge base storage: storing the generated vector data and image address information in the equipment maintenance knowledge base to facilitate subsequent retrieval and question-answering system use;

[0058] S46, based on the pre-trained multimodal large model, a multimodal equipment maintenance knowledge question-answering system is constructed by retrieval enhancement to generate RAG.

[0059] S5, retrieval and generation based on user questions. The above system extracts relevant text and image information from the knowledge base through vector retrieval technology based on user questions, combines context prompt words into a multimodal large model, and generates answer content containing answers and reference knowledge indexes.

[0060] Figure 3 This is a flowchart of using a trained multimodal large model for intelligent question answering, which specifically includes the following steps:

[0061] S51, Question Parsing, Text Vectorization, and Text Retrieval

[0062] Question analysis: After the user asks a question, the multimodal equipment maintenance knowledge question-answering system constructed in step S4 extracts text content from the question;

[0063] Text vectorization: The extracted text is input into the embedding model for vectorization processing to generate a text vector representation of the question.

[0064] Text retrieval: Use the text vectors generated above to retrieve text, tables, and image addresses related to the user’s question from the vector knowledge base.

[0065] S52, context prompt word generation: filling the retrieved text and table into the context prompt word template to obtain the filled context prompt words.

[0066] S53, input prompt word integration: integrate the context prompt words and the images and texts in the user's question to form the final input prompt words, which are input into the pre-trained multimodal large model to generate the answer content.

[0067] S54, answer generation: control the multimodal large model through prompt words to generate content containing answers and referenced knowledge index numbers.

[0068] S55, answer output and knowledge index output.

[0069] Answer output: The system outputs the generated answer to the user.

[0070] Knowledge index output: Output the referenced text knowledge index and the images contained in the knowledge index content.

[0071] S56, conversation continuation judgment: judge whether the user needs to continue the conversation, if so, continue the conversation, if not, end the process.

[0072] The present invention collects image and text data related to power plant equipment and produces a fine-tuning data set in the form of image and text pairs, so that the model can understand and process image and text information at the same time. This solves the problem that the traditional question-answering system is trained and generated based on single-modal data (such as plain text), which makes it difficult to fully utilize the image and text information related to power generation equipment, thereby affecting the accuracy and practicality of the question-answering system. It improves the model's ability to understand multimodal data and enhances the accuracy and practicality of the model when processing complex problems involving images.

[0073] The present invention also generates more accurate and comprehensive answers by combining user questions with relevant information in the vector knowledge base through the retrieval-augmented generation (RAG) method. This solves the problem that the traditional question-answering system lacks effective retrieval and utilization of external knowledge when generating answers, resulting in insufficient accuracy and reliability of the answers. This improves the accuracy and reliability of the answers generated by the system, allowing the system to provide more comprehensive and well-founded answers.

[0074] At the same time, the present invention outputs the referenced text knowledge index and the image information it contains while generating the answer, providing more intuitive and detailed reference materials. This solves the problem that the equipment maintenance question-and-answer system constructed by the traditional method is difficult to effectively quote and display specific information in multimodal content (such as equipment maintenance guidance diagrams, equipment structure diagrams, etc.) when answering questions. It enhances the intuitiveness and referenceability of the answer, allowing users to more comprehensively understand and use the information in the answer.

[0075] The present invention also discloses a computer-readable storage medium and a computer system. Specifically, a computer program is stored on the computer-readable storage medium, and after the computer program is run, the method as described above is executed. The computer system includes a processor and a storage medium, and the storage medium stores the computer program. The processor reads and runs the computer program from the storage medium to execute the method as described above.

[0076] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0077] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0078] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0079] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0080] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

[0081] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal question-answering method for power generation equipment based on retrieval enhancement generation technology, characterized in that: The following steps are involved: S1, collects graphic and text data related to power plant equipment; S2, data cleaning and labeling; S3, based on the LoRA algorithm and the data set processed in step S2, fine-tune the multimodal large model, and obtain a multimodal large model suitable for the power generation equipment question-answering scenario by observing the test data set; S4, based on the pre-trained multimodal large model, a multimodal equipment maintenance knowledge question-answering system is constructed by retrieval enhancement to generate RAG; S5, retrieval and generation based on user questions. The above system extracts relevant text and image information from the knowledge base through vector retrieval technology based on user questions, combines context prompt words into a multimodal large model, and generates answer content containing answers and reference knowledge indexes.

2. The method according to claim 1, characterized in that: The graphic data in step S1 include equipment operation and maintenance procedures, equipment factory report, technical test report, maintenance report, research report, and on-site equipment images.

3. The method according to claim 1, characterized in that Step S2 is specifically as follows: remove noise, duplicate data and irrelevant information through data cleaning, and produce a fine-tuning dataset in the form of image-text pairs through manual annotation, where the fine-tuning dataset is in the form of one image corresponding to one multi-round conversation record, providing high-quality training data for subsequent model fine-tuning.

4. The method according to claim 1, characterized in that Step S4: The specific steps of constructing a multi-modal equipment maintenance knowledge question-answering system include: S41, the system receives an input document; S42, according to the file type of the document, the system selects a corresponding parser to convert the document content into two main forms: text and image; S43, further processing the parsed text content and converting it into a vector representation; S44, writing the storage address of the image on the server into the metadata portion of the corresponding vector to ensure the association between the image and the text vector; S45, storing the generated vector data and image address information in the equipment maintenance knowledge base for subsequent retrieval and use; S46, based on the pre-trained multimodal large model, a multimodal equipment maintenance knowledge question-answering system is constructed by retrieval enhancement to generate RAG.

5. The method according to claim 4, characterized in that: The document formats in step S41 include txt, markdown, docx and pdf; and the corresponding parsers in step S42 include xt file parser, markdown file parser, docx file parser and pdf file parser.

6. The method according to claim 1, characterized in that Step S5 specifically includes the following steps: S51, starting from the user asking a question, extracting the text content in the user's question; inputting the extracted text into the embedding model, and retrieving the text, table and image address related to the user's question from the vector knowledge base through the text vector; S52, filling the retrieved text and table into the context prompt word template to obtain the filled context prompt words; S53, integrating the context prompt words and the image and text in the user's question to form a final input prompt word, which is input into a pre-trained multimodal large model to generate answer content; S54, controlling the multimodal large model to generate content including answers and referenced knowledge index numbers through prompt words; S55, outputting the generated answer and the referenced text knowledge index and the included images; S56, determining whether the user needs to continue the conversation, if so, continuing the conversation, if not, ending the process.

7. A computer-readable storage medium, characterized in that: A computer program is stored on the medium, and after the computer program is run, the method according to any one of claims 1 to 6 is executed.

8. A computer system, characterized in that: The method comprises a processor and a storage medium, wherein a computer program is stored in the storage medium, and the processor reads and runs the computer program from the storage medium to execute the method as claimed in any one of claims 1 to 6.

Images