LLM-driven internal and external knowledge dynamic fusion hybrid retrieval enhanced generation method and storage medium

Abstract

The invention provides an LLM-driven internal and external knowledge dynamic fusion hybrid retrieval enhanced generation method and a storage medium, an RAG database is constructed to generate external knowledge documents, the external knowledge documents are screened through SPT and PIT tags, the screened external knowledge documents are integrated with internal knowledge documents generated by LLM, and the integrated knowledge documents are obtained. And clustering consistent information, separating conflict information to form a knowledge document set, inputting the knowledge document set into LLM, and generating an answer.

Description

Technical Field

[0001] The present application belongs to the technical field of large language models, and specifically relates to a hybrid retrieval enhancement generation method and storage medium for dynamic fusion of internal and external knowledge driven by LLM. Background Art

[0002] The knowledge training cycle of large language models (LLMs) is long and the training cost is high. Large language models will produce hallucinations during the reasoning process. In some application scenarios of large language models, the reasoning accuracy of large language models cannot meet expectations due to the inability to obtain sufficient training samples for model training and fine-tuning.

[0003] RAG, whose full name is Retrieval Enhanced Generation in Chinese, can assist large language models in acquiring knowledge and making inferences. By constructing a retrieval enhanced generation strategy, the required rule knowledge is combined with the large language model, which enables the large language model to dynamically retrieve useful information when answering relevant questions, thereby effectively alleviating the "hallucination" problem in the reasoning process. However, the application of traditional RAG in specific professional fields requires full integration of relevant knowledge in the field. Different fields have different requirements for the integration of internal and external knowledge. If the internal and external knowledge cannot be accurately integrated, the large language model cannot respond accurately. Summary of the invention

[0004] The present application provides a hybrid retrieval enhancement generation method based on LLM-driven dynamic fusion of internal and external knowledge, which can accurately integrate internal and external knowledge and enable the large language model to make accurate responses.

[0005] The technical solution of this application is as follows: A hybrid retrieval enhancement generation method based on LLM-driven dynamic fusion of internal and external knowledge includes the following steps: S01) Build RAG database; S02) Use the word embedding model to vectorize the query question, calculate the cosine similarity RLV between the query question and each knowledge vector in the RAG database, set the RLV threshold and retain the knowledge documents in the RAG database that are greater than the RLV threshold; S03) generating an SPT tag and a PIT tag for the knowledge document retained in step S02), wherein the SPT tag is used to determine the support degree of the knowledge document for the query question, and the PIT tag is used to determine the source authority, content integrity and timeliness of the knowledge document; S04) extracting an external knowledge document set according to the SPT tag level and the PIT tag calculation value; S05) inputting the external knowledge document set and query question of step S04) into the LLM, determining the maximum number of iterations and the maximum number of generated paragraphs of the LLM, and constructing an internal knowledge paragraph prompt word template; S06) Input the query question in LLM and call the internal knowledge paragraph prompt word template to generate an internal knowledge document set; S07) performing an XOR operation on the external knowledge document set and the internal knowledge document set to obtain an initial source knowledge set, adding a source tag for distinguishing internal knowledge from external knowledge to the knowledge documents in the initial source knowledge set, and adding a PIT tag to the internal knowledge documents; S08) Use LLM to perform clustering consistent information operation on the initial source knowledge set, and perform conflict information separation operation again based on the clustering consistent information operation result; for the separated conflict information, filter the conflict information according to the PIT tag calculation value, information source, and context matching degree in turn to form a knowledge document set; S09) Input query questions and knowledge document collection into LLM and output answers.

[0006] Further, in step S03), the levels of the SPT tag include fully supported, partially supported, and not supported; Among them, full support means that the knowledge document contains keywords that directly match the query question; partial support means that the knowledge document contains concept descriptions related to the query question; no support means that the knowledge document is irrelevant to the query question.

[0007] Furthermore, in steps S03) and S07), the PIT tag determines the source authority of the knowledge document by assigning points to the knowledge source. The method for assigning points to the source authority of the knowledge document is as follows: When the source of knowledge is the latest national regulations, 5 points are awarded; When the source of knowledge is a technical manual or technical guide, 4 points are assigned; When the knowledge source is historical operation ticket records, the score is 3 points; When the knowledge source is a third-party document from a non-governmental department, 2 points are assigned; When the knowledge source is temporary log records of control business (dispatching duty log), 1 point is assigned; In steps S03) and S07), the PIT tag determines the content completeness of the knowledge document by using the word embedding model to determine whether the knowledge document covers all the steps in the query question. If yes, a score of 1 is assigned, otherwise no score is assigned; In steps S03) and S07), the PIT tag determines the timeliness of the knowledge document by judging the release date of the knowledge document. If the release date is within 1 year, a score of 1 is assigned; if the release date is more than 3 years, a score of -1 is assigned.

[0008] Further, in steps S04) and S08),; The calculation method of the PIT tag calculation value is: ; The calculated value of the PIT tag is rounded up.

[0009] Further, in step S04), the SPT tag of the external knowledge document set is fully supported or partially supported, the PIT tag calculation value is greater than 3, and the number of knowledge documents in the external knowledge document set is greater than or equal to 3.

[0010] Further, when the number of knowledge documents in the external knowledge document set is less than 3, the RLV threshold is adjusted downward based on 0.1 in step S02) until the number of knowledge documents is greater than or equal to 3.

[0011] Furthermore, in step S06), a PIT tag calculation value of 5 is assigned to the generated internal knowledge document set.

[0012] Furthermore, in step S08), irrelevant information is first filtered according to the calculated value of the PIT tag. When the calculated values ​​of the PIT tags of the information are equal, the knowledge documents whose source tags are internal knowledge are preferentially retained. When the source tags still cannot filter out irrelevant information, LLM is used to filter out irrelevant information according to context matching.

[0013] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned LLM-driven internal and external knowledge dynamic fusion hybrid retrieval enhancement generation method.

[0014] Due to the adoption of the above technical solution, the beneficial effects of this application are as follows: 1. The hybrid RAG proposed in this application realizes the automated evaluation of external knowledge. By analyzing the reliability of LLM's internal knowledge and the external knowledge of the RAG database, the two are organically integrated to form a knowledge document collection, making the information documents based on which LLM answers questions more accurate and reasonable, significantly improving the "illusion" problem of LLM, and thus outputting more accurate and reliable answers.

[0015] 2. This application screens knowledge documents through RLV thresholds and PIT tag calculation values. In the PIT tag, full attention is paid to the authority, completeness, and timeliness of the knowledge source, and the screening conditions are further deepened, which helps to improve the quality of the generated internal and external knowledge documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0017] Figure 1 The overall flow chart of a hybrid retrieval enhancement generation method based on LLM-driven dynamic fusion of internal and external knowledge provided in this application; Figure 2 A diagram of the generation process of a collection of external knowledge documents; Figure 3 This application provides an application flowchart of a hybrid retrieval enhancement generation method based on LLM-driven dynamic fusion of internal and external knowledge. DETAILED DESCRIPTION

[0018] Based on the background technology, in order to solve the hallucination problem of large language models in the process of question output and improve the quality of its answer output, refer to the attached Figure 1 and attached Figure 3 The present application provides a hybrid retrieval enhancement generation method based on LLM-driven dynamic fusion of internal and external knowledge, including the following steps: S01) Build the RAG database.

[0019] The traditional construction process of the RAG database includes data preparation, data processing, and rule knowledge vector embedding.

[0020] Taking the dispatch of power grid operation tickets as an example, data preparation is to clean up the text data of dispatch professional field operation specifications, business rules, relevant regulations and expert experience, unify the text format, and delete redundant information. Data processing is due to the length limit of large language models for input data. Too long data can easily cause the focus to be dispersed. Therefore, it is necessary to split long documents into short text blocks and maintain the continuity between the context blocks.

[0021] In this embodiment, the rule knowledge vector embedding adopts the pre-trained word embedding model Sentence-BERT, maps all processed rule experience data knowledge into high-dimensional vectors, and then stores them in the knowledge vector database. SBERT is an improved model based on BERT, which is specifically used to generate high-quality sentence-level semantic embedding vectors, so that each sentence can be independently encoded into a vector. Sentences S1 and S2 are input into the BERT model with the same parameters, and the fixed-dimensional feature vector u of sentence S1 and the fixed-dimensional feature vector v of sentence S2 are extracted by pooling, and then the cosine similarity is used to calculate the similarity between u and v, thereby solving the efficiency problem of traditional BERT in sentence pair tasks (such as semantic similarity calculation).

[0022] After the RAG database is established: S02) Use the word embedding model to vectorize the query question, calculate the cosine similarity RLV between the query question and each knowledge vector in the RAG database, set the RLV threshold and retain the knowledge documents in the RAG database that are greater than the RLV threshold.

[0023] In this embodiment, SBERT is used to vectorize the query question, and the cosine similarity RLV between the query question and each knowledge vector in the RAG database is calculated, and only knowledge documents with RLV>0.8 are retained.

[0024] S03) Generate SPT tags and PIT tags for the knowledge documents retained in step S02), wherein the SPT tags are used to determine the support level of the knowledge documents for the query questions, and the PIT tags are used to determine the source authority, content integrity and timeliness of the knowledge documents.

[0025] See attached Figure 2 ,The generation of SPT tags requires semantic matching with the help of SBERT model, and matching keywords through regular expressions. In this embodiment, the levels of SPT tags include full support, partial support, and unsupported; Among them, full support means that the knowledge document contains keywords that directly match the query question; partial support means that the knowledge document contains concept descriptions related to the query question; and no support means that the knowledge document is irrelevant to the query question. For example, "prohibited" + "grounding switch" + "close switch" are keywords that directly match the query question "Can the switch be operated when the line grounding switch is in the closed position?", and the keywords that are fully supported should also be consistent with the query question. Partial support, for example, the sentence: "Power outage requires electrical testing is an important step", which mentions the description in the query question "What type of electrical tester is needed for electrical testing when the line is out of power?" but does not explain the specific related operations. In steps S03) and S07), the PIT tag determines the source authority of the knowledge document by assigning points to the knowledge source. The method for assigning points to the source authority of the knowledge document is as follows: When the source of knowledge is the latest national regulations, 5 points are awarded; When the source of knowledge is a technical manual or technical guide, 4 points are assigned; When the knowledge source is historical operation ticket records, the score is 3 points; When the knowledge source is a third-party document from a non-governmental department, 2 points are assigned; When the knowledge source is temporary log records of regulation business, 1 point is assigned; The temporary log of control business is the dispatch duty log, which is usually recorded by staff.

[0026] In step S03) and the following S07), the PIT tag determines the content completeness of the knowledge document by using the word embedding model to determine whether the knowledge document covers all the steps in the query question. If so, a score of 1 is assigned, otherwise, no score is assigned; In steps S03) and S07), the PIT tag determines the timeliness of the knowledge document by judging the release date of the knowledge document. If the release date is within 1 year, a score of 1 is assigned; if the release date is more than 3 years, a score of -1 is assigned.

[0027] S04) extracting an external knowledge document set according to the SPT tag level and the PIT tag calculation value.

[0028] Step S04) and the following S08); The calculation method of the PIT tag calculation value is: ; The calculated value of the PIT tag is rounded up.

[0029] The calculated value through the PIT tag is equivalent to determining the priority of the knowledge document. According to the PIT calculated value, the highest PIT calculated value obtained through calculation is 4 points.

[0030] In step S04), the SPT tag of the external knowledge document set is fully supported or partially supported, the PIT tag calculation value is greater than 3, and the number of knowledge documents in the external knowledge document set is greater than or equal to 3.

[0031] When the number of knowledge documents in the external knowledge document set is less than 3, the RLV threshold is adjusted downward by 0.1 in step S02 until the number of knowledge documents is greater than or equal to 3.

[0032] S05) Input the external knowledge document set and query question of step S04) into LLM, determine the maximum number of iterations and the maximum number of generated paragraphs of LLM, and construct an internal knowledge paragraph prompt word template.

[0033] The prompt word template constructed in this embodiment is as follows: You are an operation ticket analysis assistant. According to the content of the operation event, extract the key line, station, and switch information according to the [Extraction Requirements] and [Precautions], and return it in a format, and give the inference rules based on it.

[0034]

Extraction requirements

Notes

[0035] S06) Input the query question in LLM and call the internal knowledge paragraph prompt word template to generate an internal knowledge document set.

[0036] In step S06), the PIT tag calculation value assigned to the generated internal knowledge document set is 5. According to the above, the maximum value of the PIT tag calculation value obtained by calculation is 5, so the internal knowledge document set here has the highest priority.

[0037] S07) Perform an XOR operation on the external knowledge document set and the internal knowledge document set to obtain an initial source knowledge set, add a source tag for distinguishing internal knowledge from external knowledge to the knowledge document in the initial source knowledge set, and add a PIT tag to the internal knowledge document. The PIT tag in step S07) is consistent with the PIT tag assignment rule in S03). The internal knowledge document set and the external knowledge document set are integrated through step S07).

[0038] S08) Use LLM to perform clustering consistent information operation on the initial source knowledge set, and perform separation of conflicting information operation again based on the clustering consistent information operation result; for the separated conflicting information, filter the conflicting information according to the PIT tag calculation value, information source, and context matching degree in turn to form a knowledge document set.

[0039] In the clustering consistent information operation, SBERT is used to calculate the similarity between the initial source knowledge document sets, and the calculation result information is input into LLM to assist LLM in clustering knowledge documents. When clustering, the prompt word template can be used. Clustering takes the query question, the current document set, and the source label as input, and outputs the summary of each cluster. The specific steps are to combine the similarity calculation results between documents to group the knowledge documents with the same content into the same group; generate a summary description for each group of knowledge documents. For example: Document 1: "To test the voltage, a contact tester of the corresponding voltage level must be used."; Document 2: "The electroscope must be tested and qualified."; Document 3: "Electrical testing is a mandatory step."; The generated summary is: "Electricity testing must be done using a qualified tester.".

[0040] After clustering is completed, conflicting knowledge from different sources needs to be separated. It takes the query question and the clustered documents as input and outputs a conflict group list, which contains the conflict group documents and their descriptions.

[0041] For example: Document A: "Electrical testing is a mandatory step."; Document B: "You can directly hang the ground wire without testing the electricity."; Conflict group 1: Document A, Document B; Description: "Is electrical testing a mandatory step?"; Input the conflict-separated documents and remove the information irrelevant to the current query. Then filter the irrelevant information based on the PIT tag calculation value. When the PIT tag calculation values ​​of the information are equal, the knowledge documents with the source tag of internal knowledge are retained first. When the source tag still cannot filter out irrelevant information, use LLM to filter irrelevant information based on context matching.

[0042] S09) Input query questions and knowledge document collection into LLM and output answers.

[0043] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned LLM-driven internal and external knowledge dynamic fusion hybrid retrieval enhancement generation method.

[0044] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0045] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0046] Anything not described in this application can be achieved by adopting or drawing on existing technologies.

[0047] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A hybrid retrieval enhancement generation method based on LLM-driven dynamic fusion of internal and external knowledge, characterized in that: The following steps are involved: S01) Build RAG database; S02) Use the word embedding model to vectorize the query question, calculate the cosine similarity RLV between the query question and each knowledge vector in the RAG database, set the RLV threshold and retain the knowledge documents in the RAG database that are greater than the RLV threshold; S03) generating an SPT tag and a PIT tag for the knowledge document retained in step S02), wherein the SPT tag is used to determine the support degree of the knowledge document for the query question, and the PIT tag is used to determine the source authority, content integrity and timeliness of the knowledge document; S04) extracting an external knowledge document set based on the SPT tag level and the PIT tag calculation value; S05) inputting the external knowledge document set and query question of step S04) into the LLM, determining the maximum number of iterations and the maximum number of generated paragraphs of the LLM, and constructing an internal knowledge paragraph prompt word template; S06) Input the query question in LLM and call the internal knowledge paragraph prompt word template to generate an internal knowledge document set; S07) performing an XOR operation on the external knowledge document set and the internal knowledge document set to obtain an initial source knowledge set, adding a source tag for distinguishing internal knowledge from external knowledge to the knowledge documents in the initial source knowledge set, and adding a PIT tag to the internal knowledge documents; S08) Use LLM to perform clustering consistent information operation on the initial source knowledge set, and perform conflict information separation operation again based on the clustering consistent information operation result; for the separated conflict information, filter the conflict information according to the PIT tag calculation value, information source, and context matching degree in turn to form a knowledge document set; S09) Input query questions and knowledge document collection into LLM and output answers.

2. According to claim 1, a LLM-driven internal and external knowledge dynamic fusion hybrid retrieval enhanced generation method is characterized by: In step S03), the levels of the SPT tag include fully supported, partially supported, and not supported; Among them, full support means that the knowledge document contains keywords that directly match the query question; partial support means that the knowledge document contains concept descriptions related to the query question; no support means that the knowledge document is irrelevant to the query question.

3. According to claim 2, a LLM-driven internal and external knowledge dynamic fusion hybrid retrieval enhancement generation method is characterized in that: In steps S03) and S07), the PIT tag determines the source authority of the knowledge document by assigning points to the knowledge source. The method for assigning points to the source authority of the knowledge document is as follows: When the source of knowledge is the latest national regulations, 5 points are awarded; When the source of knowledge is a technical manual or technical guide, 4 points are assigned; When the knowledge source is historical operation ticket records, the score is 3 points; When the knowledge source is a third-party document from a non-governmental department, 2 points are assigned; When the knowledge source is temporary log records of regulation business, 1 point is assigned; In steps S03) and S07), the PIT tag determines the content completeness of the knowledge document by using the word embedding model to determine whether the knowledge document covers all the steps in the query question. If yes, a score of 1 is assigned, otherwise no score is assigned; In steps S03) and S07), the PIT tag determines the timeliness of the knowledge document by judging the release date of the knowledge document. If the release date is within 1 year, a score of 1 is assigned; if the release date is more than 3 years, a score of -1 is assigned.

4. According to claim 3, a LLM-driven internal and external knowledge dynamic fusion hybrid retrieval enhancement generation method is characterized in that: In steps S04) and S08), The calculation method of the PIT tag calculation value is: ； The calculated value of the PIT tag is rounded up.

5. According to claim 4, a LLM-driven internal and external knowledge dynamic fusion hybrid retrieval enhancement generation method is characterized in that: In step S04), the SPT tag of the external knowledge document set is fully supported or partially supported, the PIT tag calculation value is greater than 3, and the number of knowledge documents in the external knowledge document set is greater than or equal to 3.

6. According to claim 5, a LLM-driven internal and external knowledge dynamic fusion hybrid retrieval enhancement generation method is characterized in that: When the number of knowledge documents in the external knowledge document set is less than 3, the RLV threshold is adjusted downward by 0.1 in step S02 until the number of knowledge documents is greater than or equal to 3.

7. The LLM-driven internal and external knowledge dynamic fusion hybrid retrieval enhancement generation method according to claim 6 is characterized in that: In step S06), a PIT tag calculation value of 5 is assigned to the generated internal knowledge document set.

8. According to claim 7, a LLM-driven internal and external knowledge dynamic fusion hybrid retrieval enhancement generation method is characterized in that: In step S08), irrelevant information is first filtered according to the calculated value of the PIT tag. When the calculated values ​​of the PIT tags of the information are equal, the knowledge documents whose source tags are internal knowledge are preferentially retained. When the source tags still cannot filter out irrelevant information, LLM is used to filter out irrelevant information according to the context matching degree.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the method as claimed in claim 8 is implemented.

Images