LLM driving-based internal and external knowledge dynamic fusion hybrid retrieval enhancement generation method and storage medium

Abstract

The application provides an LLM-driven internal and external knowledge dynamic fusion hybrid retrieval enhancement generation method and a storage medium. An RAG database is constructed to generate external knowledge documents. The external knowledge documents are filtered through SPT and PIT tags. The filtered external knowledge documents and internal knowledge documents generated by LLM are integrated. Consistent information is clustered, and conflicting information is separated to form a knowledge document set. The knowledge document set is input into LLM to generate an answer.

Description

Technical Field

[0001] This application belongs to the field of large language model technology, specifically involving an LLM-driven method for dynamic fusion and hybrid retrieval enhancement of internal and external knowledge, and its storage medium. Background Technology

[0002] Large Language Models (LLMs) have long knowledge training cycles and high training costs, and they can exhibit illusions during reasoning. In some application scenarios, the inability to obtain sufficient training samples for model training and fine-tuning leads to the reasoning accuracy of LLMs falling short of expectations.

[0003] RAG, short for Retrieval Augmentation Generative Architecture, assists large language models in acquiring knowledge and performing reasoning. By constructing retrieval augmentation strategies, the necessary rule knowledge is combined with the large language model, enabling it to dynamically retrieve useful information when answering relevant questions, thus effectively mitigating the "illusion" problem in the reasoning process. However, the application of traditional RAG in specific professional fields requires a full integration of relevant knowledge within that field. Different fields have different needs for integrating internal and external knowledge; if internal and external knowledge cannot be accurately integrated, the large language model cannot make an accurate response. Summary of the Invention

[0004] This application provides an LLM-driven method for dynamic fusion and hybrid retrieval enhancement of internal and external knowledge, which can accurately integrate internal and external knowledge and enable large language models to make accurate responses.

[0005] The technical solution of this application is as follows:

[0006] A hybrid retrieval and enhancement generation method based on LLM-driven dynamic fusion of internal and external knowledge includes the following steps:

[0007] S01) Construct the RAG database;

[0008] S02) Use word embedding model to vectorize the query question, calculate the cosine similarity RLV between the query question and each knowledge vector in the RAG database, set the RLV threshold and retain knowledge documents in the RAG database that are greater than the RLV threshold;

[0009] S03) Generate SPT tags and PIT tags for the knowledge documents retained in step S02). The SPT tags are used to determine the degree to which the knowledge documents support the query question, and the PIT tags are used to determine the source authority, content completeness and timeliness of the knowledge documents.

[0010] S04) Extract the set of external knowledge documents based on the SPT tag level and PIT tag calculated value;

[0011] S05) Input the set of external knowledge documents and query questions from step S04) into the LLM, determine the maximum number of iterations and the maximum number of paragraphs generated in the LLM, and construct the internal knowledge paragraph prompt word template;

[0012] S06) Enter the query question in LLM and call the internal knowledge paragraph prompt word template to generate an internal knowledge document collection;

[0013] S07) Perform an XOR operation on the external knowledge document set and the internal knowledge document set to obtain the initial source knowledge set. Add source tags to the knowledge documents in the initial source knowledge set to distinguish between internal and external knowledge, and add PIT tags to the internal knowledge documents.

[0014] S08) Use LLM to perform clustering to achieve consistent information on the initial source knowledge set, and then perform a separation of conflicting information based on the results of the clustering to achieve consistent information. For the separated conflicting information, filter the conflicting information according to the PIT tag value, information source, and context matching degree in turn to form a knowledge document set.

[0015] S09) Input the query question and knowledge document set into the LLM and output the answer.

[0016] Furthermore, in step S03), the SPT tag levels include fully supported, partially supported, and not supported;

[0017] Among them, it fully supports indicating that the knowledge document contains keywords that directly match the query question; partially supports indicating that the knowledge document contains conceptual descriptions related to the query question; and does not support indicating that the knowledge document is unrelated to the query question.

[0018] Furthermore, in steps S03) and S07), the PIT tag determines the source authority of the knowledge document by assigning a score to the knowledge source. The scoring method for the source authority of the knowledge document is as follows:

[0019] When the source of knowledge is the latest national regulations, it is awarded 5 points;

[0020] When the knowledge source is a technical manual or technical guide, the score is 4 points;

[0021] When the source of knowledge is historical operation ticket records, the score is 3 points;

[0022] When the knowledge source is a third-party document from a non-governmental organization, 2 points are awarded.

[0023] When the source of knowledge is temporary log records of the control business (schedule duty log), a score of 1 point is assigned;

[0024] In steps S03) and S07), the PIT tag uses a word embedding model to determine whether the knowledge document covers all the steps in the query question to determine the completeness of the knowledge document content. If yes, it is assigned a score of 1; otherwise, it is not assigned a score.

[0025] In steps S03) and S07), the PIT tag determines the timeliness of the knowledge document by judging the publication date of the knowledge document. If the publication date is within 1 year, it is assigned a score of 1; if it is more than 3 years, it is assigned a score of -1.

[0026] Furthermore, in steps S04) and S08), ;

[0027] The method for calculating the PIT tag value is as follows:

[0028] ;

[0029] The calculated value of the PIT label is rounded up.

[0030] Further, in step S04), the SPT tag of the external knowledge document set is fully supported or partially supported, the PIT tag calculation value is greater than 3, and the number of knowledge documents in the external knowledge document set is greater than or equal to 3.

[0031] Furthermore, when the number of knowledge documents in the external knowledge document set is less than 3, the RLV threshold is lowered by 0.1 in step S02) until the number of knowledge documents is greater than or equal to 3.

[0032] Further, in step S06), the PIT tag calculation value assigned to the generated internal knowledge document set is 5.

[0033] Further, in step S08), irrelevant information is first filtered based on the calculated PIT tag value. When the calculated PIT tag values ​​of the information are equal, knowledge documents with source tags as internal knowledge are retained first. When the source tags still cannot filter irrelevant information, LLM is used to filter irrelevant information based on context matching degree.

[0034] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the LLM-driven dynamic fusion and hybrid retrieval enhancement generation method for internal and external knowledge as described above.

[0035] Due to the adoption of the above technical solution, the beneficial effects of this application are as follows:

[0036] 1. The hybrid RAG proposed in this application realizes the automated evaluation of external knowledge. By analyzing the reliability of the LLM's internal knowledge and the external knowledge in the RAG database, the two are organically integrated to form a knowledge document set, making the information documents on which the LLM answers questions more accurate and reasonable, significantly improving the "illusion" problem of the LLM, and thus outputting more accurate and reliable answers.

[0037] 2. This application uses RLV threshold and PIT tag calculation value to screen knowledge documents. In the PIT tag, it pays full attention to the authority, completeness and timeliness of the knowledge source, further deepening the screening conditions and helping to improve the quality of the generated internal and external knowledge documents. Attached Figure Description

[0038] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0039] Figure 1 The overall flowchart of an LLM-driven internal and external knowledge dynamic fusion hybrid retrieval enhancement generation method provided in this application;

[0040] Figure 2 A diagram illustrating the generation process of an external knowledge document collection;

[0041] Figure 3 The flowchart of an LLM-driven dynamic fusion and hybrid retrieval enhancement generation method for internal and external knowledge provided in this application is shown. Detailed Implementation

[0042] Based on the background technology, in order to solve the illusion problem in the question output process of large language models and improve the quality of their answer output, please refer to the appendix. Figure 1 and attached Figure 3 This application provides an LLM-driven method for dynamically fusing and retrieving enhanced internal and external knowledge, comprising the following steps:

[0043] S01) Construct the RAG database.

[0044] The traditional construction process of a RAG database includes data preparation, data processing, and rule knowledge vector embedding.

[0045] Taking power grid operation ticket dispatch as an example, data preparation involves cleaning textual data such as dispatching professional operation specifications, business rules, relevant regulations, and expert experience, unifying text formats, and deleting redundant information. Data processing is necessary because large language models have length limitations on input data; excessively long data can easily lead to a dispersion of the focus of attention. Therefore, long documents need to be divided into short text blocks while maintaining the continuity between contextual blocks.

[0046] In this embodiment, the rule knowledge vector embedding adopts the pre-trained word embedding model Sentence-BERT, which maps all processed rule experience data knowledge into high-dimensional vectors and then stores them in a knowledge vector database. SBERT is an improved model based on BERT, specifically designed to generate high-quality sentence-level semantic embedding vectors, allowing each sentence to be independently encoded into a vector. Sentences S1 and S2 are input into a BERT model with the same parameters. Pooling is used to extract the fixed-dimensional feature vector u of sentence S1 and the fixed-dimensional feature vector v of sentence S2. Then, cosine similarity is used to calculate the similarity between u and v, thereby solving the efficiency problem of traditional BERT in sentence pair tasks (such as semantic similarity calculation).

[0047] After the RAG database is created:

[0048] S02) The query question is vectorized using a word embedding model. The cosine similarity (RLV) between the query question and each knowledge vector in the RAG database is calculated. An RLV threshold is set, and knowledge documents in the RAG database that are greater than the RLV threshold are retained.

[0049] In this embodiment, SBERT is used to vectorize the query question, and the cosine similarity RLV between the query question and each knowledge vector in the RAG database is calculated. Only knowledge documents with RLV > 0.8 are retained.

[0050] S03) Generate SPT tags and PIT tags for the knowledge documents retained in step S02). The SPT tags are used to determine the degree to which the knowledge documents support the query question, and the PIT tags are used to determine the source authority, content completeness and timeliness of the knowledge documents.

[0051] See appendix Figure 2 The generation of SPT tags requires semantic matching using the SBERT model, which matches keywords through regular expressions. In this embodiment, the SPT tag levels include fully supported, partially supported, and not supported.

[0052] Among these, full support indicates that the knowledge document contains keywords that directly match the query question; partial support indicates that the knowledge document contains conceptual descriptions related to the query question; and no support indicates that the knowledge document is unrelated to the query question. For example, "prohibit + "grounding switch + "close switch" is a keyword that directly matches the query question "Can the switch be operated when the grounding switch of the line is in the closed position?", and the matched keywords should also be consistent with the query question. Partial support includes statements such as "Power testing is an important step during power outages," which mentions the description in the query question "What type of voltage tester should be used for voltage testing when the line is de-energized?" but does not specify the relevant operations. In steps S03) and S07), the PIT tag determines the source authority of the knowledge document by assigning a score to the knowledge source. The scoring method for the source authority of the knowledge document is as follows:

[0053] When the source of knowledge is the latest national regulations, it is awarded 5 points;

[0054] When the knowledge source is a technical manual or technical guide, the score is 4 points;

[0055] When the source of knowledge is historical operation ticket records, the score is 3 points;

[0056] When the knowledge source is a third-party document from a non-governmental organization, 2 points are awarded.

[0057] When the source of knowledge is temporary log records of regulatory business, a score of 1 point is assigned;

[0058] The temporary log for control operations, also known as the dispatch duty log, is generally recorded by staff.

[0059] In step S03) and S07 below, the PIT tag uses a word embedding model to determine whether the knowledge document covers all the steps in the query question to determine the completeness of the knowledge document content. If yes, it is assigned a score of 1; otherwise, it is not assigned a score.

[0060] In steps S03) and S07), the PIT tag determines the timeliness of the knowledge document by judging the publication date of the knowledge document. If the publication date is within 1 year, it is assigned a score of 1; if it is more than 3 years, it is assigned a score of -1.

[0061] S04) Extract the set of external knowledge documents based on the SPT tag level and PIT tag calculated value.

[0062] In step S04) and S08 below;

[0063] The method for calculating the PIT tag value is as follows:

[0064] ;

[0065] The calculated value of the PIT label is rounded up.

[0066] The PIT value is used to determine the priority of the knowledge document. According to the PIT value, the highest PIT value obtained through calculation is 4 points.

[0067] In step S04), the SPT tag of the external knowledge document set is fully supported or partially supported, the PIT tag calculation value is greater than 3, and the number of knowledge documents in the external knowledge document set is greater than or equal to 3.

[0068] When the number of knowledge documents in the external knowledge document set is less than 3, in step S02), the RLV threshold is lowered by 0.1 as a baseline until the number of knowledge documents is greater than or equal to 3.

[0069] S05) Input the external knowledge document set and query question from step S04) into the LLM, determine the maximum number of iterations and the maximum number of paragraphs generated in the LLM, and construct the internal knowledge paragraph prompt word template.

[0070] The prompt word template constructed in this embodiment is as follows: "You are an operation ticket analysis assistant. Based on the operation event content, extract key route, station, and switch information according to the [Extraction Requirements] and [Precautions], and return it in the specified format, while also providing the inference rules used in the process."

[0071] [Extraction Requirements]

[0072] a. Objects of operation: lines, stations, switches, and high-voltage reactors, including their voltage levels, types, and numbers;

[0073] b. Equipment status: Status of lines, sites, and high-voltage reactors, including operation, hot standby, cold standby, and maintenance;

[0074]

Precautions

[0075] a. If the internal rules on which the basis are based are uncertain, it is not necessary to return to intermediate rules;

[0076] b. Multiple switches at the same station or line have combined outputs;

[0077] c. If there is no operand, then "operation object" returns NONE;

[0078] d. Each generated knowledge segment must contain different information.

[0079] S06) Enter the query question in LLM and call the internal knowledge paragraph prompt word template to generate an internal knowledge document collection.

[0080] In step S06), the PIT tag calculation value assigned to the generated internal knowledge document set is 5. As mentioned above, the maximum calculated PIT tag value is 5, therefore, the internal knowledge document set here has the highest priority.

[0081] S07) Perform an XOR operation on the external knowledge document set and the internal knowledge document set to obtain an initial source knowledge set. Add source tags to the knowledge documents in the initial source knowledge set to distinguish between internal and external knowledge, and add PIT tags to the internal knowledge documents. The PIT tags in step S07) follow the same rules as those in S03). Step S07) integrates the internal and external knowledge document sets.

[0082] S08) Use LLM to perform clustering consistency information operation on the initial source knowledge set, and then perform conflict information separation operation based on the results of the clustering consistency information operation; for the separated conflict information, filter the conflict information according to the PIT tag value, information source, and context matching degree in turn to form a knowledge document set.

[0083] In the clustering consistency information operation, SBERT is used to calculate the similarity between the initial source knowledge document sets, and the calculation results are input into the LLM to assist the LLM in knowledge document clustering. During clustering, prompt word templates can be used. Clustering takes the query question, the current document set, and source tags as input, and outputs a summary for each cluster. Specifically, the steps are: combining the document similarity calculation results to group knowledge documents with consistent content into the same group; and generating a summary description for each group of knowledge documents.

[0084] For example: Document 1: "Voltage testing requires the use of a contact voltage detector of the appropriate voltage rating."

[0085] Document 2: "The electroscope must pass inspection."

[0086] Document 3: "Voltage testing is a mandatory procedure."

[0087] The generated summary would be: "Electrical testing must be performed using a qualified electrical tester."

[0088] After clustering is completed, conflicting knowledge from different sources needs to be separated. This process takes the query question and the clustered documents as input and outputs a list of conflict groups, which contains the conflict group documents and their descriptions.

[0089] For example: Document A: "Voltage testing is a mandatory step."

[0090] Document B: "Ground wire can be directly connected without voltage testing."

[0091] Conflict Group 1: Document A, Document B; Description: "Is voltage testing a mandatory procedure?"

[0092] After inputting the conflict-separated documents, remove information irrelevant to the current query. Then, filter irrelevant information based on the calculated PIT tag value. When the calculated PIT tag values ​​of information are equal, prioritize retaining knowledge documents whose source tags are internal knowledge. If the source tags still cannot filter irrelevant information, use LLM to filter irrelevant information based on context matching degree.

[0093] S09) Input the query question and knowledge document set into the LLM and output the answer.

[0094] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to execute the aforementioned LLM-driven method for dynamic fusion and hybrid retrieval enhancement of internal and external knowledge.

[0095] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0096] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0097] For any parts not mentioned in this application, existing technologies may be used or referenced.

[0098] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for enhancing and generating internal and external knowledge through dynamic fusion and hybrid retrieval based on LLM-driven methods, characterized in that... Includes the following steps: S01) Construct the RAG database; S02) Use word embedding model to vectorize the query question, calculate the cosine similarity RLV between the query question and each knowledge vector in the RAG database, set the RLV threshold and retain knowledge documents in the RAG database that are greater than the RLV threshold; S03) Generate SPT tags and PIT tags for the knowledge documents retained in step S02). The SPT tags are used to determine the degree to which the knowledge documents support the query question, and the PIT tags are used to determine the source authority, content completeness and timeliness of the knowledge documents. S04) Extract the set of external knowledge documents based on the SPT tag level and PIT tag calculated value; S05) Input the set of external knowledge documents and query questions from step S04) into the LLM, determine the maximum number of iterations and the maximum number of paragraphs generated in the LLM, and construct the internal knowledge paragraph prompt word template; S06) Enter the query question in LLM and call the internal knowledge paragraph prompt word template to generate an internal knowledge document collection; S07) Perform an XOR operation on the external knowledge document set and the internal knowledge document set to obtain the initial source knowledge set. Add source tags to the knowledge documents in the initial source knowledge set to distinguish between internal and external knowledge, and add PIT tags to the internal knowledge documents. S08) Use LLM to perform clustering to achieve consistent information on the initial source knowledge set, and then perform a separation of conflicting information based on the results of the clustering to achieve consistent information. For the separated conflicting information, filter the conflicting information according to the PIT tag value, information source, and context matching degree in turn to form a knowledge document set. S09) Input the query question and knowledge document set into the LLM and output the answer.

2. The method for dynamic fusion and hybrid retrieval enhancement generation of internal and external knowledge based on LLM-driven approach according to claim 1, characterized in that, In step S03), the SPT tag levels include fully supported, partially supported, and not supported; Among them, it fully supports indicating that the knowledge document contains keywords that directly match the query question; partially supports indicating that the knowledge document contains conceptual descriptions related to the query question; and does not support indicating that the knowledge document is unrelated to the query question.

3. The method for dynamic fusion and hybrid retrieval enhancement generation of internal and external knowledge based on LLM-driven approach according to claim 2, characterized in that, In steps S03) and S07), the PIT tag determines the source authority of a knowledge document by assigning a score to the knowledge source. The scoring method for the source authority of a knowledge document is as follows: When the source of knowledge is the latest national regulations, it is awarded 5 points; When the knowledge source is a technical manual or technical guide, the score is 4 points; When the source of knowledge is historical operation ticket records, the score is 3 points; When the knowledge source is a third-party document from a non-governmental organization, 2 points are awarded. When the source of knowledge is temporary log records of regulatory business, a score of 1 point is assigned; In steps S03) and S07), the PIT tag uses a word embedding model to determine whether the knowledge document covers all the steps in the query question to determine the completeness of the knowledge document content. If yes, it is assigned a score of 1; otherwise, it is not assigned a score. In steps S03) and S07), the PIT tag determines the timeliness of the knowledge document by judging the publication date of the knowledge document. If the publication date is within 1 year, it is assigned a score of 1; if it is more than 3 years, it is assigned a score of -1.

4. The method for dynamic fusion and hybrid retrieval enhancement generation of internal and external knowledge based on LLM-driven approach according to claim 3, characterized in that, In steps S04) and S08), The method for calculating the PIT tag value is as follows: ； The calculated value of the PIT label is rounded up.

5. The method for dynamic fusion and hybrid retrieval enhancement generation of internal and external knowledge based on LLM-driven approach according to claim 4, characterized in that, In step S04), the SPT tag of the external knowledge document set is fully supported or partially supported, the PIT tag calculation value is greater than 3, and the number of knowledge documents in the external knowledge document set is greater than or equal to 3.

6. The method for dynamic fusion and hybrid retrieval enhancement generation of internal and external knowledge based on LLM-driven approach according to claim 5, characterized in that, When the number of knowledge documents in the external knowledge document set is less than 3, in step S02), the RLV threshold is lowered by 0.1 as a baseline until the number of knowledge documents is greater than or equal to 3.

7. The method for dynamic fusion and hybrid retrieval enhancement generation of internal and external knowledge based on LLM-driven approach according to claim 6, characterized in that, In step S06), the PIT tag calculation value assigned to the generated internal knowledge document set is 5.

8. The method for dynamic fusion and hybrid retrieval enhancement generation of internal and external knowledge based on LLM-driven approach according to claim 7, characterized in that, In step S08), irrelevant information is first filtered based on the calculated PIT tag value. When the calculated PIT tag values ​​of the information are equal, knowledge documents with source tags as internal knowledge are retained first. When the source tags still cannot filter irrelevant information, LLM is used to filter irrelevant information based on context matching degree.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in claim 8 above.

Images