Large language model multi-modal retrieval enhanced question and answer method based on equipment operation and maintenance document

Abstract

The invention belongs to the field of industry, and particularly relates to a large language model multi-modal retrieval enhanced question and answer method based on an equipment operation and maintenance document. The method comprises the following steps of: 1, converting a PDF document of equipment operation and maintenance into a document in a Markdown format; 2, generating multi-modal image subtitles on the basis of the markdown format document in the step 1; 3, firstly performing knowledge analysis based on the multi-modal image subtitles generated in the step 2, and then performing vectorization coding and similarity retrieval; and 4, by utilizing the multi-modal image subtitles generated in the step 2 and through the vectorization coding and similarity retrieval in the step 3, assisting the generation of text questions and answers based on the large language model, and realizing the large language model multi-modal retrieval enhanced questions and answers based on the equipment operation and maintenance document. The method is used for solving the problem of information analysis and understanding of various modalities aiming at equipment operation and maintenance documents which are high in speciality.

Description

TECHNICAL FIELD

[0001] The application belongs to the industrial field, and particularly relates to a large language model multi-modal retrieval enhanced question and answer method based on equipment operation and maintenance documents. BACKGROUND

[0002] Document question and answer technology is widely used in various fields of life, such as industry, medicine, education, etc. Efficient and accurate intelligent document question and answer technology can help practitioners quickly analyze the current state, decompose complex tasks, and make quick reactions and judgments. The present application focuses on the intelligent knowledge question and answer of PDF documents based on equipment operation and maintenance in the industrial field, and uses large language model prompting engineering and retrieval enhancement generation technology to realize information retrieval and question and answer generation based on PDF documents of equipment operation and maintenance. The PDF documents of equipment operation and maintenance in the industrial field contain complex and diverse information storage forms such as images, texts and tables, and have strong professional knowledge, so they cannot directly use existing multi-modal large language models to generate accurate results. General large language models generally lack domain-specific knowledge and are prone to hallucinations, and in scenarios with high professional and accuracy requirements, it is difficult to achieve satisfactory results. Retrieval enhancement generation technology provides accurate external knowledge representation by retrieving external knowledge sources, which can alleviate large model hallucinations and supplement professional domain knowledge, and is considered a method that can enhance the generation ability of general large language models in professional fields.

[0003] Equipment operation and maintenance documents contain strong professional knowledge, and contain a large amount of multi-modal information such as images, tables and texts in the documents, which are difficult for existing general large language models or multi-modal large language models to understand when solving equipment operation and maintenance problems. SUMMARY

[0004] The present application provides a large language model multi-modal retrieval enhanced question and answer method based on equipment operation and maintenance documents, which is aimed at equipment operation and maintenance documents with strong professional characteristics to solve the problem of multi-modal information analysis and understanding.

[0005] The present application provides a large language model multi-modal retrieval enhanced question and answer system based on equipment operation and maintenance documents, which is used to realize the large language model multi-modal retrieval enhanced question and answer method based on equipment operation and maintenance documents.

[0006] The present application is realized by the following technical solutions: A large language model multi-modal retrieval enhanced question and answer method based on equipment operation and maintenance documents, the method comprising the following steps: Step 1: converting the PDF document of equipment operation and maintenance into a markdown format document; Step 2: generating multi-modal image captions based on the markdown format document of step 1; Step 3: Based on the multimodal image captions generated in step 2, knowledge parsing is first performed, followed by vector encoding and similarity retrieval; Step 4: Utilize the multimodal image captions generated in step 2, and use the vectorized encoding and similarity retrieval in step 3 to assist in the generation of text question and answer based on the large language model, thereby achieving large language model multimodal retrieval enhanced question and answer based on equipment operation and maintenance documents.

[0007] Furthermore, the step 1 specifically includes dividing the information of the PDF document of equipment operation and maintenance into text information, image information and table information; For text information, directly extract and save the text, and retain the same structure and layout as the original document; For image information, it is saved as an image and index information of the image in the document is generated; For table information, save the table in image format and generate its position index.

[0008] Furthermore, the step 2 is specifically to use the context information of the paragraph where the image is located as an auxiliary model to understand the textual knowledge of the image.

[0009]

[0010] in, For image-based and contextual information Generated image captions; For the i images; Based on the image Relevant text information retrieved; It is a multimodal large language model; search () is the search function.

[0011] Furthermore, the step 3 is specifically to use the vector model BGE-M3 to extract the encoding vector of the text information,

[0012] in, BGE () represents the BGE-M3 vector model, which can encode text information into vector representation. X For the plain text information in the document, C Image caption information generated by the multimodal large language model for image information, V It is the vector representation obtained by encoding the text information in the document.

[0013] Further, the step 4 is specifically, taking the text information vector in the document as the candidate vector library, after the problem query vector is encoded into a vector using BGE-M3, top K relevant paragraphs are retrieved using cosine similarity calculation, and after relevant search information related to the problem is obtained, the answer to the problem is generated using a large language model under a designed prompt word template.

[0014] Further, the large language model is a movable type large model.

[0015] A large language model multi-modal retrieval enhanced question and answer system based on device operation and maintenance documents, the system uses the large language model multi-modal retrieval enhanced question and answer method based on device operation and maintenance documents as described above, and the system comprises: A format conversion module: converts the PDF document of device operation and maintenance into a markdown format document; A subtitle generation module: generates multi-modal image subtitles based on the markdown format document obtained by the format conversion module; A retrieval module: based on the multi-modal image subtitles generated by the subtitle generation module, knowledge analysis is performed first, and then vectorization encoding and similarity retrieval are performed; A text question and answer generation module: uses the multi-modal image subtitles generated by the subtitle generation module, and assists the generation of text questions and answers based on the large language model through the vectorization encoding and similarity retrieval of the retrieval module, to realize the large language model multi-modal retrieval enhanced question and answer based on device operation and maintenance documents.

[0016] The large language model multi-modal retrieval enhanced question and answer based on device operation and maintenance documents obtained by the above method is applied to the industrial field.

[0017] A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described above.

[0018] A computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method as described above.

[0019] The beneficial effects of the present application are: The present application uses a multi-modal large language model to process non-text information in the PDF document of device operation and maintenance, extracts image text description by generating image subtitles, and aims to extract key information contained in the image.

[0020] The present application faces query problems and adopts retrieval enhancement technology to retrieve relevant paragraphs as context information in the reconstructed PDF document information of device operation and maintenance, and enhances the effect of the large language model generating answers.

[0021] The PDF document for equipment operation and maintenance is converted into a markdown format document, and the text, picture and table information in the document can be comprehensively extracted. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a method flowchart of the present application.

[0023] Figure 2 is an industrial document analysis flowchart of the present application.

[0024] Figure 3 is a movable type language model question and answer prompt word schematic diagram of the present application. DETAILED DESCRIPTION

[0025] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0026] It should be understood that the term "comprising" as used in the specification and the appended claims indicates the presence of the recited features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0027] It should also be understood that the terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0028] The following will be described in conjunction with the drawings accompanying the present application specification Figures 1-3 The technical solutions in the embodiments of the present application are described clearly and completely, obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application.

[0029] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited by the specific embodiments disclosed below.

[0030] A large language model multi-modal retrieval enhanced question and answer method based on device operation and maintenance documents, the method comprising the following steps: Step 1: converting the PDF document of device operation and maintenance into a markdown format document; Step 2: generating multi-modal image captions based on the markdown format document of step 1; Step 3: based on the multi-modal image captions generated in step 2, first knowledge analysis, then vectorization encoding and similarity retrieval; Step 4: using the multi-modal image captions generated in step 2, and through the vectorization encoding and similarity retrieval of step 3 to assist the text question and answer generation based on large language model, realizing the large language model multi-modal retrieval enhanced question and answer based on device operation and maintenance documents.

[0031] Further, the step 1 specifically divides the information of the PDF document of device operation and maintenance into text information, picture information and table information; For text information, the text is directly extracted and saved, and the same structure layout as the original document is retained; For picture information, it is additionally saved as a picture, and index information of the picture in the document is generated; For table information, the table is saved as a picture format, and its position index is generated.

[0032] After obtaining the markdown format of the document, the subsequent retrieval enhancement generation technology is carried out by the invention with paragraphs as the basic unit.

[0033] Further, the step 2 specifically, the image caption generation technology aims to let the model understand the content in the image, which integrates computer vision and natural language processing technology. In the invention, since the operation and maintenance document contains a large amount of image, table, text and other multi-modal knowledge, in order to enhance the understanding ability of the model to image, table and other non-text knowledge, step two understands the non-text information such as image and table based on multi-modal large language model, and generates the corresponding image caption combined with the context. Specifically, when generating the image caption, the context information of the paragraph where the image is located is used as auxiliary model to understand the text knowledge of the image,

[0034]

[0035] Among them, is the image caption generated based on the image and the context information ; is the context information of the image i , which is retrieved according to the image index; ​ to retrieve relevant text information according to the image retrieved relevant text information; a multimodal large language model; search () is a retrieval function.

[0036] Further, the step 3 is specifically vectorization encoding and similarity retrieval: for text knowledge analysis, i.e. using the vector model BGE-M3 to extract the encoding vector of the text information,

[0037] wherein, BGE () represents the BGE-M3 vector model, which can encode the text information into a vector representation, X is the pure text information in the document, C is the image caption information generated by the multimodal large language model for the image information, V is the vector representation obtained by encoding the text information in the document. After obtaining the encoding vector, the cosine similarity of the candidate vector and the query vector is calculated, and the relevant paragraphs are selected as the retrieval information to assist in enhancing the large language model question answering function.

[0038] Further, the step 4 is specifically that the text information encoding vector in the document is used as the candidate vector library, the problem query vector is encoded into a vector using BGE-M3, and the top K relevant paragraphs are retrieved using cosine similarity calculation to obtain the retrieval information related to the problem. After the retrieval information related to the problem is obtained, the answer to the question is generated using the large language model under the designed prompt word template.

[0039] Further, the large language model is a living large model.

[0040] Specific implementations are as follows: One: first, an initial industrial document needs to be given, and the Mineru tool is used to preprocess the original industrial document to convert the PDF format to markdown format. Among them, for the text modality, the text information is kept unchanged; for the table and image modalities, they are kept in picture format, and the index of the picture is generated, as shown in Figure 2 .

[0041] Two: for text information, the invention is retained as candidate retrieval knowledge, and for picture information, the invention uses a multimodal large language model to generate image captions for pictures and their context paragraphs. Specifically, the two paragraphs above and below the picture are used as the context information of the picture, and the multimodal large language model is assisted to understand the image content through the prompt word engineering to generate the image caption of the corresponding picture, i.e. the text description information. As shown in Figure 2 , the "in the PDF Figure 2For the three-dimensional drawing of the water pump water turbine runner, the picture description generated by the multi-modal large language model Qwen2-VL-7B-Instruct is: "This is a three-dimensional model drawing of the runner. The upper crown and lower ring of the runner are symmetrical in position. The leak stop ring is provided to prevent water leakage. On the upper crown flange of the runner, there are exhaust holes for air exhaust after the pressure building start-up of the water pump is completed. There are also exhaust holes on the lower part of the blade and the upper part of the water discharge cone to ensure smooth air exhaust." After obtaining the image captions of all the pictures in the industrial document, the image captions replace the pictures in the original document to form a new document. The new document serves as a retrieval knowledge base and only contains text modal information. To facilitate subsequent similarity calculation, after obtaining the new document, the BGE-M3 model is used to encode the text into vector representation, and the encoding vector library of the new document is generated.

[0042] Three: After completing the industrial document preprocessing part, the user raises a question seeking an answer, and the present application calculates the similarity between the user's question and the paragraphs in the new document using cosine similarity.

[0043] For example, the user's question is: "Does the air-filled pressure water inlet pipe fill air into the runner chamber during the water pump working condition start-up?" At this time, the BGE-M3 model is used to encode the text into a vector, and then the similarity is calculated in the encoding vector library of the new document. It is worth noting that the industrial document is blocked according to paragraphs by the present application, that is, the cosine similarity between the user's question and the paragraph content is calculated, which can be specifically represented by the following formula 5-1:

[0044] Where Q is the vector of the user's question, P i is the vector of the i-th paragraph in the new document, sim i (Q, P i ) represents the calculation of the cosine similarity between the user's question and the i-th paragraph, and the final calculation obtains the top K paragraphs related to the question, K=5 in the present application, that is, the top 5 paragraphs with the highest similarity scores are retrieved. After the above process, the retrieved paragraphs are shown as follows (only the first two paragraphs are shown): The paragraph with the highest similarity score: (1) The air pressure water pipeline system water pump phase modulation or power generation phase modulation condition, its starting process uses a frequency converter (SFC) starting method, in order to reduce the starting torque, the water pressure in the runner chamber needs to be below the runner, after the unit reaches the rated speed, it can enter the water pump phase modulation or power generation phase modulation condition, then the gas is discharged, the water pump pressure process is completed, finally the unit enters the water pump pumping condition. In the design, special consideration is given to the pipeline system to realize the process of air pressure water and air pressure during water pump starting. During the trial operation of the power plant, the pipelines involved in the process are mainly adjusted and controlled to meet the requirements of the overall starting rules of the unit. The air pressure water inlet pipe: this pipe is connected to the high-pressure gas tank and connected to the draft tube pipe, during the water pump condition starting, air is filled into the runner chamber to reduce the resistance torque during starting.

[0045] The image caption with the highest similarity score: This is a three-dimensional model of a runner. The upper crown and lower ring of the runner are symmetrically positioned with leak-proof rings to prevent water leakage. On the upper crown flange of the runner, there are exhaust holes for discharging air after the water pump condition pressure starting is completed. There are also exhaust holes on the lower part of the blade and the upper part of the draft cone to ensure smooth air discharge.

[0046] Finally, the K paragraphs retrieved by the invention are input into the living large language model along with the user's question as prompt words, and the specific prompt words are as follows Figure 3 as shown; Wherein, arg.query is the user's question, context_cat is the K paragraphs retrieved, both of which are in text form input into the prompt words. The final living large language model generates an answer: "Yes, the air pressure water inlet pipe fills air into the runner chamber during water pump condition starting. This helps to reduce the resistance torque during starting and realize the process of air discharge and water pump pressure." Embodiment Two The embodiment provides a large language model multi-modal retrieval enhanced question and answer system based on equipment operation and maintenance documents, which uses the large language model multi-modal retrieval enhanced question and answer method based on equipment operation and maintenance documents as described in embodiment one, and the system comprises: A format conversion module: converts the PDF document of equipment operation and maintenance into a markdown format document; A caption generation module: generates multi-modal image captions based on the markdown format document obtained by the format conversion module; A retrieval module: based on the multi-modal image captions generated by the caption generation module, it first performs knowledge analysis, then vector encoding and similarity retrieval; The text question generation module generates a multi-modal image caption by using the multi-modal image caption generated by the caption generation module, and assists the large language model-based text question generation by using the vectorization coding and similarity retrieval of the retrieval module, so as to realize the large language model multi-modal retrieval enhanced question and answer based on the device operation and maintenance document.

[0047] Firstly, the system of the present application is aimed at the analysis of images, tables and other non-text modalities in PDF documents. The system innovatively proposes an image caption generation system based on a context-based multi-modal large language model. The context-based system can help the multi-modal large language model better understand the image scene information, which is crucial for image description generation in the industrial field. The multi-modal large language model generates an image description to replace the image data, and reconstructs the original pure text data into a new document. The system can reduce the interference caused by the large language model processing heterogeneous data, and utilize the reasoning ability of the large language model for text data to realize the device operation and maintenance document question and answer function in the industrial field.

[0048] Embodiment three The embodiment uses the large language model multi-modal retrieval enhanced question and answer based on the device operation and maintenance document obtained by the method of embodiment one in the industrial field.

[0049] Embodiment four The electronic device provided by the embodiment of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory is used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and the processor are connected through a bus. Specifically, the processor realizes any step in the above embodiment one by running the above computer program stored in the memory.

[0050] It should be understood that, in the embodiments of the present application, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0051] The memory can include read-only memory, flash memory, and random access memory, and provide instructions and data to the processor. Part or all of the memory can also include non-volatile random access memory.

[0052] As can be seen from the above, the electronic device provided by the embodiment of the present application can realize the large language model multi-modal retrieval enhanced question and answer method as described in embodiment one by running a computer program, support device operation and maintenance question and answer technology, support the analysis of multi-modal information such as images, texts, and tables in the information retrieval source, and at the same time propose a new large language model retrieval enhancement generation method. The multi-modal large language model is used to extract image captions for image and table information, forming a new text document as a retrieval information source. When facing the questions raised by the user, the large language model retrieves the relevant knowledge in the information source to enhance the question and answer ability of the large language model. By using the large language model prompt engineering and multi-modal retrieval enhancement technology, the device operation and maintenance document question and answer field in the industrial scene can provide maintenance suggestions, maintenance methods, principle analysis, etc. of the device.

[0053] It should be understood that the above integrated modules / units, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiments of the present application can also be completed by a computer program instructing related hardware, and the above-mentioned computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of the above-mentioned various method embodiments can be realized. The above-mentioned computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form, etc. The above-mentioned computer readable medium can include any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier wave signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the above-mentioned computer readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0054] The above description of disclosed implementations enables one of ordinary skill in the art to make and use the application. Various modifications to these implementations will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other implementations without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the implementations shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0055] It can be clearly understood by a person skilled in the art that, for the convenience and conciseness of description, only the division of the above functional units and modules is taken as an example, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction, and are not used to limit the protection scope of the application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0056] It should be noted that the method and its details provided by the above embodiments can be combined into the device and equipment provided by the embodiments, and mutual reference will not be described here.

[0057] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or in a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0058] In the embodiments provided by the present application, it should be understood that the disclosed device / terminal equipment and method can be implemented in other ways. For example, the above-described device / equipment embodiments are only schematic, and the division of the above modules or units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0059] The above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent ones; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A large language model multimodal retrieval enhanced question answering method based on equipment operation and maintenance documents, characterized by: The method comprises the following steps: Step 1: Convert the PDF document of equipment operation and maintenance into a markdown format document; Step 2: Generate multimodal image captions based on the markdown format document in step 1; Step 3: Based on the multimodal image captions generated in step 2, knowledge parsing is first performed, followed by vector encoding and similarity retrieval; Step 4: Utilize the multimodal image captions generated in step 2, and use the vectorized encoding and similarity retrieval in step 3 to assist in the generation of text question and answer based on the large language model, thereby achieving large language model multimodal retrieval enhanced question and answer based on equipment operation and maintenance documents.

2. The method according to claim 1, characterized in that Specifically, step 1 includes dividing the information of the PDF document of equipment operation and maintenance into text information, image information, and table information; For text information, directly extract and save the text, and retain the same structure and layout as the original document; For image information, it is saved as an image and index information of the image in the document is generated; For table information, save the table in image format and generate its position index.

3. The method according to claim 1, characterized in that Specifically, step 2 is to use the contextual information of the paragraph where the image is located as an auxiliary model to understand the textual knowledge of the image. in, For image-based and contextual information Generated image captions; For the i images; Based on the image Relevant text information retrieved; It is a multimodal large language model; search () is the search function.

4. The method according to claim 1, characterized in that Specifically, step 3 is to use the vector model BGE-M3 to extract the encoding vector of the text information. in, BGE () represents the BGE-M3 vector model, which can encode text information into vector representation. X For the plain text information in the document, C Image caption information generated by the multimodal large language model for image information, V It is the vector representation obtained by encoding the text information in the document.

5. The method according to claim 4, characterized in that: Specifically, step 4 includes using the text information encoding vector in the document as a candidate vector library, encoding the question query vector into a vector using BGE-M3, and then retrieving the top K relevant paragraphs using cosine similarity calculation. After obtaining the retrieval information related to the question, the large language model is used to generate the answer to the question under the designed prompt word template.

6. The method according to claim 5, characterized in that The large language model is a movable type large model.

7. A large language model multimodal retrieval enhanced question answering system based on equipment operation and maintenance documents, characterized by: The system uses the large language model multimodal retrieval enhanced question answering method based on equipment operation and maintenance documents as described in any one of claims 1 to 6, and the system includes: Format conversion module: converts PDF documents of equipment operation and maintenance into markdown format documents; Subtitle generation module: Generates multimodal image subtitles based on the markdown format document obtained by the format conversion module; Retrieval module: Based on the multimodal image captions generated by the caption generation module, knowledge parsing is first performed, followed by vectorized encoding and similarity retrieval; Text question and answer generation module: Utilizes the multimodal image subtitles generated by the subtitle generation module, and uses the vectorized encoding and similarity retrieval of the retrieval module to assist in the generation of text question and answer based on a large language model, thereby realizing large language model multimodal retrieval and enhanced question and answer based on equipment operation and maintenance documents.

8. A large language model multimodal retrieval enhanced question and answer system based on equipment operation and maintenance documents obtained by the method according to any one of claims 1 to 6 is applied to the industrial field.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images