Large language model and knowledge base cooperation method and device based on retrieval enhancement generation

Abstract

The invention provides a big language model and knowledge base cooperation method and device based on retrieval enhancement generation, and the method comprises the steps: obtaining a question proposed by a user, inputting the question into a general question and answer system based on a big language model, and obtaining an answer to the question; performing vectorization processing on the question, and sending the question into a knowledge base for retrieval to obtain a statement set related to the question in the knowledge base; and combining the answer and the statement set into a final answer for the question. Through cooperation of the large language model and the knowledge base, the problem that knowledge of the large language model is insufficient and lagged is solved, the risk of synthesizing wrong answers is reduced, the accuracy and context correlation of the answers are improved, and the method adapts to a unique expression mode in a specific field.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for cooperating with a large language model and a knowledge base based on retrieval enhancement generation. Background Art

[0002] At present, general-purpose question-answering systems based on LLM (large language model) (such as ChatGPT, Wenxin Yiyan and other products) are booming. However, general-purpose question-answering systems are often unable to provide answers due to lack of sufficient domain knowledge when dealing with a large number of problems in specific fields; the knowledge they contain comes from their training data, so they cannot reflect the latest domain knowledge or information; at the same time, for questions in specific fields, general-purpose question-answering systems are often prone to giving wrong answers, leading to misleading users and introducing risks. Summary of the invention

[0003] The present invention aims to provide a method and device for coordinating a large language model and a knowledge base based on retrieval enhancement generation, which overcomes the above problems or at least partially solves the above problems.

[0004] To achieve the above object, the technical solution of the present invention is specifically implemented as follows:

[0005] One aspect of the present invention provides a method for cooperating a large language model and a knowledge base based on retrieval enhancement generation, comprising:

[0006] Obtaining a question raised by a user, inputting the question into a general question-answering system based on a large language model, and obtaining an answer to the question;

[0007] Vectorizing the problem, sending the problem to a knowledge base for retrieval, and obtaining a set of statements in the knowledge base related to the problem;

[0008] The answer is combined with the set of statements into a final answer to the question.

[0009] Optionally, after the question is vectorized, it is sent to a knowledge base for retrieval, and before a set of statements related to the question in the knowledge base is obtained, the method further includes: constructing the knowledge base.

[0010] Optionally, the constructing the knowledge base includes:

[0011] The documents in the document collection are vectorized and stored in the form of triples of sentence content, order within the document, and document index;

[0012] A hierarchical clustering method is used to organize and manage the knowledge base.

[0013] Optionally, vectorizing and storing the documents in the document collection in the form of a triple of statement content, order within the document, and document index includes:

[0014] Convert the sentence content into a sentence vector;

[0015] Treat each of the sentence vectors as an independent cluster;

[0016] Use the mean of all sentence vectors in the cluster as the feature of the cluster, calculate the cosine similarity between all cluster features, and form a distance matrix;

[0017] Find the two clusters with the highest similarity in the distance matrix;

[0018] Merge the two clusters with the highest similarity into a new cluster;

[0019] Update the features of all clusters;

[0020] Return to execute the steps of finding the two clusters with the highest similarity in the distance matrix, merging the two clusters with the highest similarity into a new cluster, and updating the features of all clusters until all sentence vectors are merged into one cluster.

[0021] Optionally, vectorizing the question and sending it to a knowledge base for retrieval to obtain a set of statements related to the question in the knowledge base includes:

[0022] Vectorizing the problem to obtain a vector representation of the problem;

[0023] Calculate the cosine similarity between the problem and the left and right branch clusters of the current cluster;

[0024] Taking the branch cluster with higher cosine similarity as the current cluster;

[0025] Repeat the step of calculating the cosine similarity between the question and the left and right branch clusters of the current cluster, and taking the branch cluster with higher cosine similarity as the current cluster, until the sentence capacity of the current cluster meets the requirement;

[0026] Get the set of all statements in the current cluster.

[0027] Another aspect of the present invention provides a large language model and knowledge base collaboration device based on retrieval enhancement generation, comprising:

[0028] An answer module is used to obtain questions raised by users, input the questions into a general question-answering system based on a large language model, and obtain answers to the questions;

[0029] A retrieval module is used to vectorize the question and send it to a knowledge base for retrieval to obtain a set of statements in the knowledge base that are related to the question;

[0030] A merging module is used to merge the answer and the statement set into a final answer to the question.

[0031] Optionally, the device further comprises: a construction module, configured to construct the knowledge base after the retrieval module vectorizes the question and sends the question to the knowledge base for retrieval, and before obtaining a set of statements in the knowledge base related to the question.

[0032] Optionally, the construction module constructs the knowledge base in the following manner:

[0033] The documents in the document collection are vectorized and stored in the form of triples of sentence content, order within the document, and document index;

[0034] A hierarchical clustering method is used to organize and manage the knowledge base.

[0035] Optionally, the construction module vectorizes and stores the documents in the document collection in the form of a triple of statement content, order within the document, and document index in the following manner:

[0036] Convert the sentence content into a sentence vector;

[0037] Treat each of the sentence vectors as an independent cluster;

[0038] Use the mean of all sentence vectors in the cluster as the feature of the cluster, calculate the cosine similarity between all cluster features, and form a distance matrix;

[0039] Find the two clusters with the highest similarity in the distance matrix;

[0040] Merge the two clusters with the highest similarity into a new cluster;

[0041] Update the features of all clusters;

[0042] Return to execute the steps of finding the two clusters with the highest similarity in the distance matrix, merging the two clusters with the highest similarity into a new cluster, and updating the features of all clusters until all sentence vectors are merged into one cluster.

[0043] Optionally, the retrieval module vectorizes the question in the following manner, sends the question to a knowledge base for retrieval, and obtains a set of statements in the knowledge base related to the question:

[0044] Vectorizing the problem to obtain a vector representation of the problem;

[0045] Calculate the cosine similarity between the problem and the left and right branch clusters of the current cluster;

[0046] Taking the branch cluster with higher cosine similarity as the current cluster;

[0047] Repeat the step of calculating the cosine similarity between the question and the left and right branch clusters of the current cluster, and taking the branch cluster with higher cosine similarity as the current cluster, until the sentence capacity of the current cluster meets the requirement;

[0048] Get the set of all statements in the current cluster.

[0049] It can be seen that the method and device for collaborating a large language model and a knowledge base based on retrieval enhancement generation provided by the present invention can solve the problems of insufficient and lagging knowledge of the large language model through the collaboration of the large language model and the knowledge base, reduce the risk of synthesizing wrong answers, improve the accuracy and contextual relevance of the answers, and adapt to the unique expressions of specific fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0051] Figure 1 A flowchart of a method for cooperating a large language model and a knowledge base based on retrieval enhancement generation provided in an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a method for cooperating between a large language model and a knowledge base based on retrieval enhancement generation provided in an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of a hierarchical tree structure composed of sentences in a knowledge base provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of the structure of a large language model and knowledge base collaboration device based on retrieval enhancement generation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0056] Figure 1 A flowchart of a method for cooperating a large language model and a knowledge base based on retrieval enhancement generation provided by an embodiment of the present invention is shown. Figure 2 A schematic diagram of a method for cooperating a large language model and a knowledge base based on retrieval enhancement generation provided by an embodiment of the present invention is shown, see Figure 1 and Figure 2 The embodiment of the present invention provides a method for collaborating a large language model and a knowledge base based on retrieval enhancement generation, including:

[0057] S1, obtaining a question raised by a user, inputting the question into a general question-answering system based on a large language model, and obtaining an answer to the question;

[0058] S2, vectorizing the question and sending it to a knowledge base for retrieval to obtain a set of statements in the knowledge base related to the question;

[0059] S3, merging the answer with the statement set into a final answer to the question.

[0060] Specifically, the present invention first directly inputs the question Q raised by the user into the question-answering LLM to obtain the answer A to the question; at the same time, after vectorizing the question, it is sent to the knowledge base D for retrieval to obtain a set of sentences S related to the question in the knowledge base; finally, the answer of the question-answering LLM and S are combined into the final answer A to Q of the present invention. f .

[0061] As an optional implementation of an embodiment of the present invention, after the question is vectorized, it is sent into a knowledge base for retrieval, and before a set of sentences related to the question in the knowledge base is obtained, the method for collaborating with a large language model and a knowledge base based on retrieval enhancement generation provided in an embodiment of the present invention also includes: constructing the knowledge base.

[0062] As an optional implementation of the embodiment of the present invention, the building of the knowledge base includes:

[0063] The documents in the document collection are vectorized and stored in the form of triples of sentence content, order within the document, and document index;

[0064] A hierarchical clustering method is used to organize and manage the knowledge base.

[0065] As an optional implementation of the embodiment of the present invention, vectorizing and storing the documents in the document collection in the form of a triple of sentence content, order within the document, and document index includes:

[0066] Convert the sentence content into a sentence vector;

[0067] Treat each of the sentence vectors as an independent cluster;

[0068] Use the mean of all sentence vectors in the cluster as the feature of the cluster, calculate the cosine similarity between all cluster features, and form a distance matrix;

[0069] Find the two clusters with the highest similarity in the distance matrix;

[0070] Merge the two clusters with the highest similarity into a new cluster;

[0071] Update the features of all clusters;

[0072] Return to execute the steps of finding the two clusters with the highest similarity in the distance matrix, merging the two clusters with the highest similarity into a new cluster, and updating the features of all clusters until all sentence vectors are merged into one cluster.

[0073] Specifically, the knowledge base is represented by the symbol D, which means a collection of documents in a specific field, containing a total of N documents, where the i-th document is represented by D i :

[0074] D={D1,D2,...,D i ,...,D N}

[0075] In general, a document is composed of a set of ordered sentences. Therefore, there are N i A document consisting of sentences D i It can be expressed as:

[0076]

[0077] in For Document D i It should be noted that in the knowledge base involved in the present invention, each entry is stored in the form of a triple of "sentence content-order in document-document index":

[0078]

[0079] This triple form indicates that the text content The jth sentence from the i-th document. The advantage of this representation is that in the final answer A f In A, if the text in the knowledge base is cited, it can support f Explicit citations are marked in .

[0080] For The present invention first vectorizes and stores the knowledge base as a storage unit, and organizes and manages the knowledge base using a hierarchical clustering method. The present invention uses the open source tool sentence_transformers (which contains the open source all-MiniLM-L6-v2 model for converting sentences into 384-dimensional sentence vectors) to transform sentences Convert to sentence vector

[0081]

[0082] After this processing, all sentences of different lengths in the knowledge base are converted into vectors of the same dimension, where f st Represents the function of sentence_transformers. Afterwards, for all The present invention performs hierarchical clustering according to the following steps:

[0083] Step 1: Transform each sentence vector Treated as an independent cluster;

[0084] Step 2: Use the mean of all sentence vectors in the cluster as the feature of the cluster, calculate the cosine similarity between all cluster features, and form a distance matrix;

[0085] Step 3: Find the two clusters with the highest similarity in the distance matrix;

[0086] Step 4: Merge these two clusters into a new cluster;

[0087] Step 5: Update the features of all clusters;

[0088] Step 6: Repeat steps 3-5 until all sentence vectors are finally merged into one cluster.

[0089] Finally, the present invention obtains a Figure 3 The tree diagram shown in Figure 1 realizes the organization and management of all statements in the knowledge base. At the same time, after the above steps, the characteristics of each cluster are also retained. For clusters with branches, the characteristics of the left cluster in the branch are represented by c left , the feature of the right cluster in its branch is represented by c right .

[0090] As an optional implementation of the embodiment of the present invention, vectorizing the question and sending it to a knowledge base for retrieval to obtain a set of statements related to the question in the knowledge base includes:

[0091] Vectorizing the problem to obtain a vector representation of the problem;

[0092] Calculate the cosine similarity between the problem and the left and right branch clusters of the current cluster;

[0093] Taking the branch cluster with higher cosine similarity as the current cluster;

[0094] Repeat the step of calculating the cosine similarity between the question and the left and right branch clusters of the current cluster, and taking the branch cluster with higher cosine similarity as the current cluster, until the sentence capacity of the current cluster meets the requirement;

[0095] Get the set of all statements in the current cluster.

[0096] Specifically, using the above f st Vectorize the problem Q and get the vector representation v of the problem Q Q :

[0097] v Q =f st (Q)

[0098] So far, v Q , as well as Figure 3 The feature representation of each cluster in the hierarchical tree shown (c left and c righy ), these three vectors are in the same vector space and have the same vector dimension, and the similarity between them can be calculated.

[0099] For Q, the present invention uses v Q To express, and from Figure 3 The hierarchical tree shown starts from the top level and searches downwards. The search process is as follows:

[0100] Step 1: Calculate Q and the left and right branch clusters of the current cluster (c left and c right ),

[0101] Step 2: Change the current cluster to a branch cluster with a higher cosine similarity obtained in the previous step.

[0102] Step 3: Repeat steps 1-2 until the statement capacity of the current cluster meets the requirement.

[0103] Returns the set S of all the statements in the current cluster. Obviously, the statement capacity of the current cluster decreases as the search process goes deeper. Therefore, the cluster capacity when the above search process is terminated supports artificial definition in the present invention, for example, it can be set to a power of 2 such as 1, 2, 4, etc. according to actual conditions.

[0104] Finally, for question Q, the answer given by the present invention is A f =A∪S.

[0105] Among them, for two vectors X and Y, the cosine similarity between the two is:

[0106] It should be noted that the present invention processes natural language text data and does not involve the processing of modal data such as voice, image, and video. The knowledge base involved in the present invention is a collection of documents in a specific field, and the present invention only processes the text data in each document.

[0107] It can be seen that the method for collaborating with a large language model and a knowledge base based on retrieval enhancement generation provided by the present invention can solve the problems of insufficient and lagging knowledge of the large language model through the collaboration of the large language model and the knowledge base, reduce the risk of synthesizing wrong answers, improve the accuracy and contextual relevance of the answers, and adapt to the unique expressions of specific fields.

[0108] Figure 4 The schematic diagram of the structure of the large language model and knowledge base collaboration device based on retrieval enhancement generation provided by an embodiment of the present invention is shown. The large language model and knowledge base collaboration device based on retrieval enhancement generation applies the above method. The following is only a brief description of the structure of the large language model and knowledge base collaboration device based on retrieval enhancement generation. For other matters not covered, please refer to the relevant description of the large language model and knowledge base collaboration method based on retrieval enhancement generation, see Figure 4 The large language model and knowledge base collaboration device based on retrieval enhancement generation provided by the embodiment of the present invention includes:

[0109] An answer module is used to obtain questions raised by users, input the questions into a general question-answering system based on a large language model, and obtain answers to the questions;

[0110] A retrieval module is used to vectorize the question and send it to a knowledge base for retrieval to obtain a set of statements in the knowledge base that are related to the question;

[0111] A merging module is used to merge the answer and the statement set into a final answer to the question.

[0112] As an optional implementation of an embodiment of the present invention, the large language model and knowledge base collaboration device based on retrieval enhancement generation also includes: a construction module, which is used to construct the knowledge base before the retrieval module vectorizes the question and sends it to the knowledge base for retrieval, and obtains a set of sentences related to the question in the knowledge base.

[0113] As an optional implementation of the embodiment of the present invention, the construction module constructs the knowledge base in the following manner:

[0114] The documents in the document collection are vectorized and stored in the form of triples of sentence content, order within the document, and document index;

[0115] A hierarchical clustering method is used to organize and manage the knowledge base.

[0116] As an optional implementation of the embodiment of the present invention, the construction module vectorizes and stores the documents in the document collection in the form of a triple of sentence content, order within the document, and document index in the following manner:

[0117] Convert the sentence content into a sentence vector;

[0118] Treat each of the sentence vectors as an independent cluster;

[0119] Use the mean of all sentence vectors in the cluster as the feature of the cluster, calculate the cosine similarity between all cluster features, and form a distance matrix;

[0120] Find the two clusters with the highest similarity in the distance matrix;

[0121] Merge the two clusters with the highest similarity into a new cluster;

[0122] Update the features of all clusters;

[0123] Return to execute the steps of finding the two clusters with the highest similarity in the distance matrix, merging the two clusters with the highest similarity into a new cluster, and updating the features of all clusters until all sentence vectors are merged into one cluster.

[0124] As an optional implementation of the embodiment of the present invention, the retrieval module vectorizes the question in the following manner, sends the question to the knowledge base for retrieval, and obtains a set of sentences related to the question in the knowledge base:

[0125] Vectorizing the problem to obtain a vector representation of the problem;

[0126] Calculate the cosine similarity between the problem and the left and right branch clusters of the current cluster;

[0127] Taking the branch cluster with higher cosine similarity as the current cluster;

[0128] Repeat the step of calculating the cosine similarity between the question and the left and right branch clusters of the current cluster, and taking the branch cluster with higher cosine similarity as the current cluster, until the sentence capacity of the current cluster meets the requirement;

[0129] Get the set of all statements in the current cluster.

[0130] It can be seen that the large language model and knowledge base collaboration device based on retrieval enhancement generation provided by the present invention can solve the problems of insufficient and lagging knowledge of the large language model through the collaboration of the large language model and the knowledge base, reduce the risk of synthesizing wrong answers, improve the accuracy and context relevance of the answers, and adapt to the unique expressions of specific fields.

[0131] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for collaborating a large language model and a knowledge base based on retrieval enhancement generation, characterized in that: include: Obtaining a question raised by a user, inputting the question into a general question-answering system based on a large language model, and obtaining an answer to the question; Vectorizing the problem, sending the problem to a knowledge base for retrieval, and obtaining a set of statements in the knowledge base related to the problem; The answer is combined with the set of statements into a final answer to the question.

2. The method according to claim 1, characterized in that After the problem is vectorized, it is sent to a knowledge base for retrieval, and before a set of statements related to the problem in the knowledge base is obtained, the method further includes: constructing the knowledge base.

3. The method according to claim 2, characterized in that The constructing of the knowledge base comprises: The documents in the document collection are vectorized and stored in the form of triples of sentence content, order within the document, and document index; A hierarchical clustering method is used to organize and manage the knowledge base.

4. The method according to claim 3, characterized in that The vectorized storage of the documents in the document collection in the form of a triple of sentence content, order within the document, and document index includes: Convert the sentence content into a sentence vector; Treat each of the sentence vectors as an independent cluster; Use the mean of all sentence vectors in the cluster as the feature of the cluster, calculate the cosine similarity between all cluster features, and form a distance matrix; Find the two clusters with the highest similarity in the distance matrix; Merge the two clusters with the highest similarity into a new cluster; Update the features of all clusters; Return to execute the steps of finding the two clusters with the highest similarity in the distance matrix, merging the two clusters with the highest similarity into a new cluster, and updating the features of all clusters until all sentence vectors are merged into one cluster.

5. The method according to claim 4, characterized in that The problem is vectorized and sent to a knowledge base for retrieval, and a set of statements related to the problem in the knowledge base is obtained, including: Vectorizing the problem to obtain a vector representation of the problem; Calculate the cosine similarity between the problem and the left and right branch clusters of the current cluster; Taking the branch cluster with higher cosine similarity as the current cluster; Repeat the step of calculating the cosine similarity between the question and the left and right branch clusters of the current cluster, and taking the branch cluster with higher cosine similarity as the current cluster, until the sentence capacity of the current cluster meets the requirement; Gets the set of all statements in the current cluster.

6. A large language model and knowledge base collaboration device based on retrieval enhancement generation, characterized in that: include: An answer module is used to obtain questions raised by users, input the questions into a general question-answering system based on a large language model, and obtain answers to the questions; A retrieval module is used to vectorize the question and send it to a knowledge base for retrieval to obtain a set of statements in the knowledge base that are related to the question; A merging module is used to merge the answer and the statement set into a final answer to the question.

7. The device according to claim 6, characterized in that Also includes: A construction module is used to construct the knowledge base after the retrieval module vectorizes the question and sends it to the knowledge base for retrieval, and before obtaining a set of statements related to the question in the knowledge base.

8. The device according to claim 7, characterized in that The building module builds the knowledge base in the following way: The documents in the document collection are vectorized and stored in the form of triples of sentence content, order within the document, and document index; A hierarchical clustering method is used to organize and manage the knowledge base.

9. The device according to claim 8, characterized in that The construction module vectorizes and stores the documents in the document collection in the form of triples of sentence content, order within the document, and document index in the following manner: Convert the sentence content into a sentence vector; Treat each of the sentence vectors as an independent cluster; Use the mean of all sentence vectors in the cluster as the feature of the cluster, calculate the cosine similarity between all cluster features, and form a distance matrix; Find the two clusters with the highest similarity in the distance matrix; Merge the two clusters with the highest similarity into a new cluster; Update the features of all clusters; Return to execute the steps of finding the two clusters with the highest similarity in the distance matrix, merging the two clusters with the highest similarity into a new cluster, and updating the features of all clusters until all sentence vectors are merged into one cluster.

10. The device according to claim 9, characterized in that The retrieval module vectorizes the question and sends it to the knowledge base for retrieval to obtain a set of statements related to the question in the knowledge base in the following manner: Vectorizing the problem to obtain a vector representation of the problem; Calculate the cosine similarity between the problem and the left and right branch clusters of the current cluster; Taking the branch cluster with higher cosine similarity as the current cluster; Repeat the step of calculating the cosine similarity between the question and the left and right branch clusters of the current cluster, and taking the branch cluster with higher cosine similarity as the current cluster, until the sentence capacity of the current cluster meets the requirement; Get the set of all statements in the current cluster.

Images