RAG-based construction technology multi-mode intelligent knowledge base question and answer processing method, medium and equipment

Abstract

The invention provides an RAG-based construction technology multi-modal intelligent knowledge base question and answer processing method, medium and equipment, and the method comprises the steps: screening an input multi-modal text, processing various files after screening according to corresponding preset technical means, carrying out the embedding processing of obtained text segments, and inputting the text segments into a vector database; obtaining user questions, converting the questions into cue words through the large language model, and generating a query task according to the cue words; constructing a knowledge graph to respond to the questions of the user; matching the text segments in a database based on the multi-element mixing similarity; the text segments and the questions are input into a large language model according to the template, and answers are generated; and generating an answer text in a specified format. Knowledge and experience can be inherited in a natural language form, and the situation that a large amount of manpower and material resources are consumed when a knowledge base is built is avoided; the requirement on the document quality in the RAG application process can be reduced through a set multi-mode processing method; and more accurate matching results and answer output are provided.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and specifically relates to a RAG-based construction technology multimodal intelligent knowledge base question-answering processing method, medium and equipment. Background Art

[0002] RAG (Retrieval-Augmented Generation) technology is a method to enhance large language models (LLM). It reduces the problem of generating factually incorrect content by retrieving relevant documents from an external knowledge base and calculating semantic similarity. The application of RAG in question-answering systems mainly includes three steps: indexing (segmenting documents into chunks, encoding them into vectors, and storing them in a vector database), retrieval (retrieving the top k chunks most relevant to the question based on semantic similarity), and generation (inputting the original question and the retrieved chunks into the LLM together to generate the final answer).

[0003] The current RAG technology mainly performs relevant retrieval operations on texts. However, for construction and building enterprise users, especially traditional enterprises, they have a large number of historical technical documents and program files retained in other modes such as pictures and videos. Similarly, national standards and specifications are all scanned copies, which cannot be directly converted, and the document quality is not high. The current technology cannot meet the document retrieval enhancement of other modes, which limits the implementation of the RAG intelligent knowledge base question and answer system of traditional enterprises. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a method, medium and device for question-answering of a multimodal intelligent knowledge base of construction technology based on RAG, aiming to integrate the automatic creation and query capabilities of knowledge graphs and solve the problems of diverse text source modalities and poor output format restrictions in industry applications.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a RAG-based construction technology multimodal intelligent knowledge base question-answering processing method, comprising the following steps:

[0007] S1: Input a multimodal file and screen it according to the modality, embed the screened multimodal file according to the corresponding modality, and input the text segment obtained after the embedding process into the vector database;

[0008] S2: Obtain user questions and convert them into prompt words through a large language model, and generate query tasks based on the prompt words;

[0009] S3: Use natural language processing technology to perform entity recognition and relationship extraction on the text segment obtained in step S1, express it in the form of triples, and build a knowledge graph based on it; respond to the user's question in step S2 based on the constructed knowledge graph;

[0010] S4: according to the query task generated in step S2, vector search and keyword search are performed in the vector database, and the text segment for responding to the query task is screened out by combining the search results of the two;

[0011] S5: The text segments selected in step S4 and the knowledge graph response results in step S3 are input into the large language model using the prompt template arranged based on the CO-STAR principle to summarize and generate answers;

[0012] S6: Generate an answer text in a specified format based on the answer generated in step S5.

[0013] Optionally, in step S1, the multimodal file includes a picture, a PDF, a table, and a text document.

[0014] Optionally, in step S1, embedding the screened multimodal file according to the corresponding modality, and inputting the text segment obtained after the embedding process into the vector database, specifically:

[0015] For images, start OCR to extract text from the images, identify and summarize the image content, and compose the image information element text segment from the image content description and the text in the image, and embed the image information element text segment; for PDF, directly extract text or start OCR and layout recovery tools to extract text, and embed the extracted text; for tables, parse them through HTML methods, and embed the extracted text; for text documents, embed the extracted text after word segmentation according to the strategy;

[0016] Among them, after embedding processing, a key-value pair consisting of id and vector expression is formed and input into the vector database.

[0017] Optionally, in step S3, the response to the user's question in step S2 based on the constructed knowledge graph is specifically:

[0018] The target entity is extracted from the user's question, the corresponding nodes and node context relationships are retrieved in the constructed knowledge graph, and converted into natural language.

[0019] Optionally, in step S4, vector search and keyword search are performed in the vector database, and the text segment for responding to the query task is screened out by combining the search results of the two, specifically:

[0020] For vector retrieval, the Pearson correlation coefficient is used to calculate the vector similarity between the query task and the text segment in the vector database;

[0021] For keyword retrieval, the BM25 ranking function is used to re-rank the similarities of the query task and the text segments in the vector database by combining TD-IDF and document length factors, and the top-ranked similarity is selected as the keyword similarity;

[0022] According to the vector similarity S 向量 Similarity with keyword S 关键词 , calculate the comprehensive similarity S according to the following formula 综合 :

[0023] S 综合 =0.8×S 向量 +0.2×S 关键词 ;

[0024] Filter out the corresponding text segments based on the comprehensive similarity.

[0025] Optionally, in step S5, after the summary generates an answer, the answer is scored. If the score does not meet the set scoring threshold, the process returns to step S2 and regenerates the question until the set scoring threshold is met or the maximum number of repetitions is reached.

[0026] Optionally, in step S6, the answer text in a specified format is generated according to the answer generated in step S5, specifically:

[0027] The generated answer is input into the fine-tuning large model, which is a model that uses standard format data in the field of construction to fine-tune some parameters. The answer is verified and format-constrained and checked before being output to the user.

[0028] Optionally, the format constraint and check are specifically:

[0029] Convert the answer to latex format through prompt constraints, or call the word template to generate a document in a specified format based on the keywords generated by the answer.

[0030] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the RAG-based construction technology multimodal intelligent knowledge base question-answering processing method as described in the first aspect.

[0031] In a third aspect, the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the RAG-based construction technology multimodal intelligent knowledge base question and answer processing method as described in the first aspect is implemented.

[0032] The beneficial effects of the present invention are as follows: compared with the traditional knowledge storage mode, especially for industries such as the construction industry where employee mobility is extremely high, the present invention can pass on knowledge and experience through natural language dialogue, while avoiding the construction of a knowledge base that consumes a lot of manpower and material resources; by setting a multimodal processing method, the requirements for document quality in the RAG application process are reduced; by adopting a hybrid similarity retrieval, a more accurate matching result is provided; the problem that traditional industry scanned documents, pictures and other document format data cannot be directly applied is solved; by setting a prompt and configuring multiple large language models for combined use, the output of answers in a specified format is completed; combined with the large model semantic extraction technology, a knowledge graph is established and the query results are verified based on the graph relationship, and the output results are more accurate and complete in scenarios with complex knowledge bases. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The flowchart of the question-answering processing method of the RAG-based construction technology multimodal intelligent knowledge base.

[0034] Figure 2 An example diagram of answer text formatted for slot template generation.

[0035] Figure 3 This is the interface diagram for question and answer processing of RAG-based construction technology multimodal intelligent knowledge base. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0037] In one embodiment, the present invention proposes a construction technology multimodal intelligent knowledge base question-answering processing method based on RAG, such as Figure 1 As shown, the specific steps include:

[0038] S1: Input a multimodal file into a file database and screen the input multimodal file, wherein each modal file type includes: picture, PDF, table and text document.

[0039] After screening, each type of file is processed according to the corresponding preset technical means: if the file mode is a picture, OCR is started to extract the text in the picture, and the multimodal model is used to identify and summarize the picture content. The "picture content description" and "picture text content" constitute the picture information element text segment, and then the embedding model is started to vectorize the picture information element (i.e., embedding processing). The multimodal model refers to a large language model with visual modality, including LLava, GPT-4v, etc.; if the file is a PDF, try to extract the text directly. If it is a scanned PDF, start the OCR and layout recovery tool to extract the text in the PDF, and then enable the text embedding model to embed the extracted text; if the file is a table, call the table parsing tool, parse the table header, content, etc. through the html method, and then enable the text embedding model to embed the extracted text; if the file is a text document, the text embedding model is directly enabled to embed the extracted text after word segmentation according to a certain strategy. After word segmentation is completed, continue to improve the word segmentation effect, and then perform conversion optimization on text extraction and layout analysis. Layout analysis means that the extracted text layout logic is consistent with the original file.

[0040] The text segments obtained after embedding processing (each type has one) are input into the vector database, where different modalities use different database storage methods. After embedding, the text modality forms a key-value pair composed of the text segment id and the high-dimensional vector expression and inputs it into the database. The image modality stores the key-value pairs composed of the image id value, image content description, and high-dimensional vector of the text in the image, and fills them into the vector database for query.

[0041] S2: Obtain user questions and convert them into more standard and easier to search prompts through a large language model, and generate query tasks based on the prompts.

[0042] If the user's expression is not standardized, it will be transcribed through the big model and then input into the model to query the knowledge base. If it involves a database query, the database query statement will be converted into a database query through natural language, so it is divided into the user's "query" and the "task" formed after the query is converted.

[0043] S3: Build a document knowledge graph based on semantics, including using natural language processing technology to identify entities (people, machines, time, place, etc.) and extract relationships from embedded text segments, expressed in the form of triples, including three parts: subject, object and relationship. Then build a knowledge graph based on this information, graph G {kg} The nodes in the V represent entities {kg} , side E {kg}Represents the relationship between entities. Next, the knowledge graph is optimized through algorithms to ensure its accuracy and completeness, and the constructed knowledge graph is stored in the graph database. The basic formula of the graph is expressed as follows:

[0044] G {kg} =V {kg} ×E {kg}

[0045] Entity V and edge E are represented in vector space. In the TransE model, a relation (edge) is considered as an operation that transforms one entity vector to another entity vector. The following is a simplified formula that uses entity v and edge E to express this transformation relationship, where the L2 norm is used. This formula calculates the Euclidean distance between entity V_1 and entity V_2 after being transformed by relation E, and uses the negative value of this distance as the score. The lower the score, the stronger the association between entities V1 and V2 through relation E.

[0046] s(V1, V2, E) = -|V1+E-V2|2

[0047] In this embodiment, in order to ensure the accuracy of the search results, the corresponding nodes are retrieved from the comprehensively generated knowledge graph. The specific method is: extract the target entity from the user input, find the corresponding node and node context relationship in the graph, and convert it into natural language. All operations are completed by the problem processing model.

[0048] S4: Match embedded objects in the vector database based on multi-factor mixed similarity. To enhance relevance and avoid mis-checking and missing, use mixed precision for screening, comprehensively weight and sort vector similarity and text similarity, and return the top-k text segments with the highest mixed similarity as information source data after screening and sorting. The top-k sorting method is a multi-factor mixed search (keyword + vector), where the multi-factors include:

[0049] (a) Vector cosine similarity uses the Pearson correlation coefficient, which is calculated as follows:

[0050]

[0051] (b) Keyword search: Through the BM25 ranking function, TD-IDF (term frequency-inverse document frequency) and document length factors are combined to achieve the similarity re-ranking of documents and questions. The formula is as follows:

[0052]

[0053] Among them, the vector similarity weight is 0.8, the full text similarity weight is 0.2, and the comprehensive similarity formula is as follows:

[0054] S 综合=0.8×S 向量 +0.2×S 关键词

[0055] S5: Based on the text segments filtered by comprehensive similarity and the ternary relationship pairs of the knowledge graph, use the prompt template arranged based on the CO-STAR principle (context, objective, style, tone, audience, response), input the large language model used to generate answers, summarize and generate the target answer, and use the prompt design to let it score the answer based on the comprehensive question and answer. If the set scoring threshold is not met, the question is regenerated and the answer is retrieved and generated. This step is set with a maximum number of repetitions to prevent users from waiting too long.

[0056] S6: Generate an answer text in a specified format, input the answer into a fine-tuned large model with smaller parameters, where the fine-tuned large model refers to a model that uses standard format data in the field of construction to fine-tune some parameters, and verify and check the answer format, and output it to the user. Specifically, the target answer in the previous step is converted into a format such as latex through prompt constraints, or a word template is called to generate specified keywords, and a document in a specified format is generated by using a template to replace keywords, such as filling in the value of the corresponding keyword according to the construction plan template, to complete the final file output result that conforms to the user's format. Figure 2 is an example of a simple slot template.

[0057] The question-answering processing method of the multimodal intelligent knowledge base of construction technology based on RAG proposed in this embodiment aims to improve the efficiency and accuracy of user queries. It can automatically process various files such as pictures, PDFs, tables, text documents, etc., extract content through intelligent technology and convert it into vectors, use hybrid retrieval strategies to match objects, and generate intelligent answers in combination with large language models. Finally, accurate and formatted answers are output to users, which improves user experience and data processing capabilities. The question-answering processing method of the multimodal intelligent knowledge base of construction technology based on RAG can be used as follows: Figure 3 The interface shown is implemented.

[0058] In another embodiment, the present invention proposes a computer-readable storage medium storing a computer program, which enables a computer to execute the RAG-based construction technology multimodal intelligent knowledge base question-answering processing method described in the aforementioned embodiment.

[0059] In another embodiment, the present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the RAG-based construction technology multimodal intelligent knowledge base question and answer processing method described in the aforementioned embodiment is implemented.

[0060] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0061] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0062] The above are only preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A question-answering processing method for a multimodal intelligent knowledge base of construction technology based on RAG, characterized in that: The steps include: S1: Input a multimodal file and screen it according to the modality, embed the screened multimodal file according to the corresponding modality, and input the text segment obtained after the embedding process into the vector database; S2: Obtain user questions and convert them into prompt words through a large language model, and generate query tasks based on the prompt words; S3: Use natural language processing technology to perform entity recognition and relationship extraction on the text segment obtained in step S1, express it in the form of triples, and build a knowledge graph based on it; respond to the user's question in step S2 based on the constructed knowledge graph; S4: according to the query task generated in step S2, vector search and keyword search are performed in the vector database, and the text segment for responding to the query task is screened out by combining the search results of the two; S5: The text segments selected in step S4 and the knowledge graph response results in step S3 are input into the large language model using the prompt template arranged based on the CO-STAR principle to summarize and generate answers; S6: Generate an answer text in a specified format based on the answer generated in step S5.

2. The RAG-based construction technology multimodal intelligent knowledge base question-answering processing method according to claim 1, characterized in that: In step S1, the multimodal file includes pictures, PDF, tables and text documents.

3. The RAG-based construction technology multimodal intelligent knowledge base question-answering processing method according to claim 2, characterized in that: In step S1, the screened multimodal file is embedded according to the corresponding modality, and the text segment obtained after the embedding process is input into the vector database, specifically: For images, start OCR to extract text from the images, identify and summarize the image content, and compose the image information element text segment from the image content description and the text in the image, and embed the image information element text segment; for PDF, directly extract text or start OCR and layout recovery tools to extract text, and embed the extracted text; for tables, parse them through HTML methods, and embed the extracted text; for text documents, embed the extracted text after word segmentation according to the strategy; Among them, after embedding processing, a key-value pair consisting of id and vector expression is formed and input into the vector database.

4. The RAG-based construction technology multimodal intelligent knowledge base question-answering processing method according to claim 1, characterized in that: In step S3, the constructed knowledge graph responds to the user's question in step S2, specifically: The target entity is extracted from the user's question, the corresponding nodes and node context relationships are retrieved in the constructed knowledge graph, and converted into natural language.

5. The RAG-based construction technology multimodal intelligent knowledge base question-answering processing method according to claim 1, characterized in that: In step S4, vector search and keyword search are performed in the vector database, and the text segment for responding to the query task is screened out by combining the search results of the two, specifically: For vector retrieval, the Pearson correlation coefficient is used to calculate the vector similarity between the query task and the text segment in the vector database; For keyword retrieval, the BM25 ranking function is used to re-rank the similarities of the query task and the text segments in the vector database by combining TD-IDF and document length factors, and the top-ranked similarity is selected as the keyword similarity; According to the vector similarity S 向量 Similarity with keyword S 关键词 , calculate the comprehensive similarity S according to the following formula 综合 : S 综合 =0.8×S 向量 +0.2×S 关键词 ; Filter out the corresponding text segments based on the comprehensive similarity.

6. The RAG-based construction technology multimodal intelligent knowledge base question-answering processing method according to claim 1, characterized in that: In step S5, after the summary generates an answer, the answer is scored. If the score does not meet the set scoring threshold, the process returns to step S2 and regenerates the question until the set scoring threshold is met or the maximum number of repetitions is reached.

7. The RAG-based construction technology multimodal intelligent knowledge base question-answering processing method according to claim 1, characterized in that: In step S6, the answer text in a specified format is generated according to the answer generated in step S5, specifically: The generated answer is input into the fine-tuning large model, which is a model that uses standard format data in the field of construction to fine-tune some parameters. The answer is verified and format-constrained and checked before being output to the user.

8. The RAG-based construction technology multimodal intelligent knowledge base question-answering processing method according to claim 7, characterized in that: The format constraints and checks are specifically as follows: Convert the answer to latex format through prompt constraints, or call the word template to generate a document in a specified format based on the keywords generated by the answer.

9. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables the computer to execute the RAG-based construction technology multimodal intelligent knowledge base question and answer processing method as described in any one of claims 1-8.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the RAG-based construction technology multimodal intelligent knowledge base question-answering processing method as described in any one of claims 1 to 8 is implemented.

Images