Document question and answer method and system based on LLM, RAG and knowledge graph

Abstract

The invention discloses a document question and answer method and system based on LLM, RAG and a knowledge graph. The method comprises the steps that an obtained historical question set and document paragraphs related to the historical question set are classified; extracting a plurality of first keywords in each type of historical problem document set; utilizing a knowledge graph to construct a relationship between the plurality of first keywords and the document paragraphs related to the plurality of first keywords to obtain a relationship knowledge graph; constructing a keyword library containing a plurality of first keywords and words related to the first keywords; extracting a plurality of second keywords in the to-be-answered question, and searching a plurality of third keywords similar to the plurality of second keywords from a keyword library; retrieving a plurality of document paragraphs corresponding to the plurality of second keywords and the plurality of third keywords from a relational knowledge graph; and inputting the to-be-answered question and the plurality of document paragraphs into the trained prediction large model to obtain a target answer. The accuracy and efficiency of document question answering can be improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent question answering technology, and in particular to a document question answering method and system based on LLM, RAG and knowledge graph. Background Art

[0002] With the development of information technology, document question answering based on big models is becoming more and more popular. Document question answering is generally done by uploading a document. When receiving a user's question, the big model will analyze and understand the given document, and then extract the answer from multiple documents with the highest relevance to answer the question.

[0003] However, many document question answering methods and systems directly use large-scale language models (LLMs). Although large-scale language models have advantages such as simplicity and easy explanation, they often perform poorly when processing complex and nonlinear data. In addition, large-scale language models have limited capabilities. When the required information exceeds the scope of the model training data or requires the latest data, large-scale language models may not be able to provide accurate answers. Therefore, how to improve the accuracy and efficiency of document question answering is an urgent problem to be solved. Summary of the invention

[0004] This application aims to propose a document question and answer method and system based on LLM, RAG and knowledge graph, which can improve the accuracy and efficiency of document question and answer.

[0005] In a first aspect, an embodiment of the present application provides a document question-answering method based on LLM, RAG and knowledge graph, the method comprising:

[0006] Classifying the acquired historical question set and document paragraphs related to the historical question set to obtain multiple categories of historical question document sets, wherein one historical question document set includes a plurality of historical questions of the same type and document paragraphs related to the plurality of historical questions;

[0007] Extracting a plurality of first keywords from each category of the historical problem document set, wherein the plurality of first keywords are keywords selected from the historical problem document set;

[0008] Using the knowledge graph to construct the relationship between the multiple first keywords and the document paragraphs related to the multiple first keywords, to obtain a relationship knowledge graph corresponding to each type of the historical question document set;

[0009] constructing a keyword library including the plurality of first keywords and words related to the first keywords;

[0010] Extracting multiple second keywords from the question to be answered, and searching the keyword library for multiple third keywords that are close to the multiple second keywords, wherein the second keywords are keywords extracted from the question to be answered, and the third keywords are keywords searched from the keyword library;

[0011] Retrieving a plurality of document paragraphs corresponding to the plurality of second keywords and the plurality of third keywords from the relational knowledge graph using a RAG method;

[0012] The question to be answered and the multiple document paragraphs are input into the trained prediction model to obtain the target answer to the question to be answered.

[0013] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0014] This method obtains multiple categories of historical question document sets by classifying the acquired historical question sets and the document paragraphs related to the historical question sets, extracts multiple first keywords in each category of historical question document sets, and uses the knowledge graph to construct the relationship between the multiple first keywords and the document paragraphs related to the multiple first keywords, thereby obtaining the relationship knowledge graph corresponding to each category of historical question document sets. By classifying the acquired historical question sets and the document paragraphs related to the historical question sets, and then constructing the knowledge graph of each category of historical question document sets, it is possible to reduce the retrieval time for the later stage, thereby improving the efficiency of document question answering. Then, a keyword library containing multiple first keywords and words related to the first keywords is constructed, multiple second keywords in the questions to be answered are extracted, and multiple third keywords similar to the multiple second keywords are searched from the keyword library. By searching for words related to the keywords, the probability of obtaining an incorrect answer due to the inability to find the same keyword in the relationship knowledge graph for the keywords of the questions raised by the user can be reduced, and relatively correct answers can be obtained through similar words. The RAG method is used to retrieve multiple document paragraphs corresponding to multiple second keywords and multiple third keywords from the relational knowledge graph. Multiple document paragraphs corresponding to the keywords can be quickly and accurately obtained from the relational knowledge graph, and the questions to be answered and multiple document paragraphs are input into the trained prediction large model to obtain the target answers to the questions to be answered. By combining the large model, the relational knowledge graph and the RAG method, it is only necessary to find the answers in multiple very relevant document paragraphs, which can improve the accuracy and efficiency of document question and answer.

[0015] In some implementations, the acquired historical question set and document paragraphs related to the historical question set are classified to obtain multiple categories of historical question document sets, including:

[0016] A vector space model is used to construct a context feature vector for each of the historical issues;

[0017] Calculating the similarity between the context feature vectors corresponding to the two historical questions to obtain context similarity;

[0018] Using a word vector model to construct a semantic feature vector for each of the historical questions;

[0019] Calculating the similarity between the semantic feature vectors corresponding to the two historical questions to obtain semantic similarity;

[0020] Constructing a fitness function according to the context similarity and the semantic similarity;

[0021] Based on the fitness function, a genetic algorithm is used to classify each historical question in the acquired historical question set and document paragraphs related to the historical question set to obtain a multi-category historical question document set.

[0022] In some implementations, extracting a plurality of first keywords from each type of the historical question document set includes:

[0023] Using a word segmenter to segment each historical issue in each category of the historical issue document set to obtain a first word segmentation set;

[0024] Using a word segmenter to segment the document paragraphs related to each of the historical issues to obtain a second word segmentation set;

[0025] According to the first segmented word set and the second segmented word set, a keyword extraction method is adopted to extract a plurality of first keywords in the first segmented word set.

[0026] In some implementations, the step of constructing a keyword library comprising the plurality of first keywords and words related to the first keywords comprises:

[0027] Combining a plurality of first keywords in the historical question document set of all categories into a keyword set;

[0028] Find at least one of a synonym and a near synonym for each keyword in the keyword set;

[0029] Each keyword and at least one of a synonym and a near-synonym corresponding to each keyword are combined into a keyword library.

[0030] In some implementations, searching the keyword library for a plurality of third keywords that are close to the plurality of second keywords includes:

[0031] Using a word vector model to vectorize each of the second keywords to obtain multiple first keyword vectors;

[0032] Using a word vector model to vectorize each keyword in the keyword library to obtain multiple second keyword vectors;

[0033] Calculating the similarity between each of the first keyword vectors and each of the second keyword vectors to obtain multiple similarity values;

[0034] Sorting the multiple similarity values ​​to obtain sorted similarity values;

[0035] A preset number of similarity values ​​are selected from the sorted similarity values, and keywords corresponding to the selected preset number of similarity values ​​are used as third keywords.

[0036] In some embodiments, the trained prediction model is obtained by:

[0037] Based on the LLM model, a large prediction model is constructed;

[0038] The prediction big model is trained using the acquired several historical issues, document paragraphs related to the historical issues, and the relational knowledge graph to obtain a trained prediction big model.

[0039] In some implementations, after inputting the question to be answered and the multiple document paragraphs into the trained prediction model to obtain the target answer to the question to be answered, the document question answering method based on LLM, RAG and knowledge graph further includes:

[0040] Inputting the question to be answered and the multiple document paragraphs into the trained prediction model to predict the target answer to the question to be answered;

[0041] Recording the target answer and the unanswered question corresponding to the target answer as historical records;

[0042] The historical record is used to update the historical question set and the document paragraphs related to the historical question set.

[0043] In a second aspect, the embodiment of the present application further provides a document question-answering system based on LLM, RAG and knowledge graph, the system comprising:

[0044] A data classification unit is used to classify the acquired historical question set and the document paragraphs related to the historical question set to obtain multiple categories of historical question document sets, wherein one historical question document set includes a plurality of historical questions of the same type and document paragraphs related to the plurality of historical questions;

[0045] A keyword extraction unit, used to extract a plurality of first keywords from each type of the historical problem document set, wherein the plurality of first keywords are keywords selected from the historical problem document set;

[0046] A knowledge graph construction unit, configured to use the knowledge graph to construct the relationship between the plurality of first keywords and the document paragraphs related to the plurality of first keywords, and obtain a relationship knowledge graph corresponding to each type of the historical question document set;

[0047] A keyword library construction unit, used to construct a keyword library including the plurality of first keywords and words related to the first keywords;

[0048] A keyword search unit, used to extract a plurality of second keywords from the question to be answered, and search for a plurality of third keywords close to the plurality of second keywords from the keyword library, wherein the second keywords are keywords extracted from the question to be answered, and the third keywords are keywords searched from the keyword library;

[0049] A document paragraph retrieval unit, configured to retrieve a plurality of document paragraphs corresponding to the plurality of second keywords and the plurality of third keywords from the relational knowledge graph using a RAG method;

[0050] The target answer prediction unit is used to input the question to be answered and the multiple document paragraphs into the trained prediction model to obtain the target answer to the question to be answered.

[0051] In the third aspect, an embodiment of the present application also provides an electronic device, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a document question and answer method based on LLM, RAG and knowledge graph as described above.

[0052] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute a document question-answering method based on LLM, RAG and knowledge graph as described above.

[0053] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0055] Figure 1 It is a flowchart of an embodiment of a document question-answering method based on LLM, RAG and knowledge graph provided by the present application;

[0056] Figure 2 It is a structural diagram of an embodiment of a document question-answering system based on LLM, RAG and knowledge graph provided by the present application;

[0057] Figure 3 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application. DETAILED DESCRIPTION

[0058] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0059] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0060] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0061] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0062] With the development of information technology, document question answering based on big models is becoming more and more popular. Document question answering is generally done by uploading a document. When receiving a user's question, the big model will analyze and understand the given document, and then extract the answer from multiple documents with the highest relevance to answer the question.

[0063] First, some nouns involved in this application are analyzed:

[0064] Large Language Model (LLM): refers to a deep learning model trained with large amounts of text data that can generate natural language text or understand the meaning of language text.

[0065] Retrieval-Augmented Generation (RAG): is a model that combines retrieval and generation techniques. It generates answers or content by referencing information from an external knowledge base. It has strong interpretability and customization capabilities and is suitable for multiple natural language processing tasks such as question-answering systems, document generation, and intelligent assistants.

[0066] Knowledge graph: It is called knowledge domain visualization or knowledge domain mapping map in the library and information industry. It is a series of various graphics that show the development process and structural relationship of knowledge. It uses visualization technology to describe knowledge resources and their carriers, and to mine, analyze, construct, draw and display knowledge and their interrelationships.

[0067] TF-IDF algorithm: is a statistical method used to evaluate the importance of a word to a document set or a document in a corpus.

[0068] Genetic algorithm: It is a computational model of biological evolution that simulates the natural selection and genetic mechanisms of Darwin's theory of biological evolution. It is a method of searching for the optimal solution by simulating the natural evolution process.

[0069] However, many document question answering methods and systems directly use large-scale language models (LLMs). Although large-scale language models have advantages such as simplicity and easy explanation, they often perform poorly when processing complex and nonlinear data. In addition, large-scale language models have limited capabilities. When the required information exceeds the scope of the model training data or requires the latest data, large-scale language models may not be able to provide accurate answers. Therefore, how to improve the accuracy and efficiency of document question answering is an urgent problem to be solved.

[0070] In order to solve the above-mentioned problem of how to improve the accuracy and efficiency of document question answering, this application proposes a document question answering method and system based on LLM, RAG and knowledge graph.

[0071] Reference Figure 1 , the embodiment of the present application provides a document question answering method based on LLM, RAG and knowledge graph, and the method comprises the following steps:

[0072] Step S100: classify the acquired historical question set and the document paragraphs related to the historical question set to obtain multiple categories of historical question document sets, wherein one historical question document set includes a plurality of historical questions of the same type and document paragraphs related to the plurality of historical questions;

[0073] Step S200: extracting a plurality of first keywords from each type of historical problem document set, where the plurality of first keywords are keywords selected from the historical problem document set;

[0074] Step S300: Use the knowledge graph to construct the relationship between the first keywords and the document paragraphs related to the first keywords, and obtain the relationship knowledge graph corresponding to each type of historical problem document set;

[0075] Step S400: construct a keyword library including a plurality of first keywords and words related to the first keywords;

[0076] Step S500: extract multiple second keywords from the question to be answered, and search for multiple third keywords similar to the multiple second keywords from the keyword library, where the second keywords are keywords extracted from the question to be answered, and the third keywords are keywords searched from the keyword library;

[0077] Step S600: Retrieve multiple document paragraphs corresponding to multiple second keywords and multiple third keywords from the relational knowledge graph using the RAG method;

[0078] Step S700: input the question to be answered and multiple document paragraphs into the trained prediction model to obtain the target answer to the question to be answered.

[0079] In this embodiment, by classifying the acquired historical question set and the document paragraphs related to the historical question set, multiple categories of historical question document sets are obtained, multiple first keywords in each category of historical question document sets are extracted, and the relationship between multiple first keywords and document paragraphs related to multiple first keywords is constructed using the knowledge graph to obtain the relationship knowledge graph corresponding to each category of historical question document sets. By classifying the acquired historical question set and the document paragraphs related to the historical question set, and then constructing the knowledge graph of each category of historical question document sets, the retrieval time can be reduced in the later stage, thereby improving the efficiency of document question answering. Then, a keyword library containing multiple first keywords and words related to the first keywords is constructed, multiple second keywords in the questions to be answered are extracted, and multiple third keywords similar to the multiple second keywords are searched from the keyword library. By searching for words related to keywords, the probability of obtaining wrong answers due to the inability to find the same keywords in the relationship knowledge graph for the keywords of the questions raised by the user can be reduced, and relatively correct answers can be obtained through similar words. The RAG method is used to retrieve multiple document paragraphs corresponding to multiple second keywords and multiple third keywords from the relational knowledge graph. It is possible to quickly and accurately obtain multiple document paragraphs corresponding to the keywords from the relational knowledge graph, input the question to be answered and the multiple document paragraphs into the trained prediction large model, and obtain the target answer to the question to be answered. By combining the large model, the relational knowledge graph and the RAG method, it is only necessary to find the answer in multiple very relevant document paragraphs, which can improve the accuracy and efficiency of document question and answer.

[0080] The above classification of the acquired historical question set and the document paragraphs related to the historical question set to obtain multiple categories of historical question document sets can be carried out by using clustering methods or other neural network methods in the prior art to classify the acquired historical question set and the document paragraphs related to the historical question set.

[0081] The above-mentioned historical question set may be a set of questions in a professional field or a set of questions in a non-professional field.

[0082] The above-mentioned document paragraphs related to the historical question set are document paragraphs after the historical questions are answered, and can be regarded as answer documents.

[0083] The above-mentioned extraction of multiple first keywords in each category of historical problem document collection may be performed by using a TF-IDF algorithm to extract multiple first keywords in each category of historical problem document collection, or may be performed by using other methods in the prior art to extract multiple first keywords in each category of historical problem document collection.

[0084] The above-mentioned use of knowledge graph to construct the relationship between multiple first keywords and document paragraphs related to the multiple first keywords. Since the first keywords are extracted based on document paragraphs, each keyword has at least one related document paragraph corresponding to it. The use of knowledge graph can construct the relationship between keywords and document paragraphs.

[0085] The above-mentioned knowledge graph can be called knowledge domain visualization or knowledge domain mapping map in the library and information industry. It is a series of various graphics that show the knowledge development process and structural relationship. It uses visualization technology to describe knowledge resources and their carriers, and mine, analyze, construct, draw and display knowledge and their interrelationships.

[0086] Knowledge graph is a modern theory that achieves the purpose of multidisciplinary integration by combining the theories and methods of applied mathematics, graphics, information visualization technology, information science and other disciplines with metrological citation analysis, co-occurrence analysis and other methods, and using visual graphs to vividly display the core structure, development history, frontier fields and overall knowledge architecture of the discipline.

[0087] The above-mentioned construction of the keyword library including the plurality of first keywords and words related to the first keywords may be manually constructing the keyword library including the plurality of first keywords and words related to the first keywords through human experience.

[0088] The above-mentioned words related to the first keyword may be synonyms, near-synonyms, or words associated with the first keyword.

[0089] The above-mentioned searching for multiple third keywords similar to multiple second keywords from the keyword library may be to use similarity calculation to calculate the similarity between each keyword in the keyword library and each second keyword, and then select multiple relatively similar keywords as third keywords.

[0090] In the above, multiple third keywords that are close to multiple second keywords are searched from the keyword library. Since the keyword library is constructed by multiple first keywords and words related to the first keywords, the third keyword is equivalent to the first keyword and the words related to the first keyword. The purpose of expanding keywords is to prevent the second keyword extracted from the question to be answered from being the same or related to the first keyword, thereby finding inaccurate related document paragraphs. This third keyword is also close to or related to multiple second keywords extracted from the question to be answered. Therefore, this third keyword can be used to find more accurate document paragraphs associated with the second keyword in the later stage, and finally a more accurate answer can be obtained by predicting the training of the large model, which solves the problem that the large language model may not be able to provide accurate answers when the required information exceeds the model training data range or the latest data is required.

[0091] The above-mentioned RAG method (Retrieval-Augmented Generation) can be a new framework that combines the two machine learning methods of retrieval and generation.

[0092] The above-mentioned large prediction model can be a large language model (LLM model), or a model that combines the LLM model with a support vector machine. Combining the LLM model with the support vector machine can prevent the large language model from overfitting, making the target answer of the document question and answer more accurate.

[0093] The technical solution of this embodiment can be applied to a variety of document question-and-answer scenarios, such as education, search engines, and corporate knowledge bases. By using knowledge graphs and automatically updating historical question sets and document paragraphs related to historical question sets, the automation and intelligence of knowledge entry into the database is realized, reducing the workload of manual combing and annotation, and improving the efficiency of entry into the database. A structured and graphed internal enterprise knowledge base is constructed by comprehensively adopting knowledge graphs, RAG methods, and predictive big models. Knowledge is closely related to each other, achieving efficient organization and storage, and providing users with natural language question-and-answer services, breaking away from the limitations of traditional keyword retrieval, and greatly improving the accuracy and convenience of knowledge access.

[0094] In some implementations, the acquired historical question set and document paragraphs related to the historical question set are classified to obtain multiple categories of historical question document sets, including:

[0095] The vector space model is used to construct the context feature vector of each historical question;

[0096] Calculate the similarity between the context feature vectors corresponding to the two historical questions to obtain the context similarity;

[0097] Use the word embedding model to construct the semantic feature vector of each historical question;

[0098] Calculate the similarity between the semantic feature vectors corresponding to the two historical questions to obtain the semantic similarity;

[0099] Construct a fitness function based on contextual similarity and semantic similarity;

[0100] Based on the fitness function, a genetic algorithm is used to classify each historical question in the acquired historical question set and the document paragraphs related to the historical question set, and a multi-category historical question document set is obtained.

[0101] In this embodiment, by calculating the similarity between the context feature vectors corresponding to two historical questions, and calculating the similarity between the semantic feature vectors corresponding to the two historical questions, and then comprehensively considering the context similarity and semantic similarity, constructing a fitness function, and finally using a genetic algorithm for classification based on the fitness function, the accuracy of classification can be improved.

[0102] The above vector space model can simplify the text content into vectors in the vector space.

[0103] The above-mentioned use of the word vector model to construct the semantic feature vector of each historical question can be to map words into real vectors, which can capture the semantic relationship between words, thereby constructing each historical question into a semantic feature vector. The word vector model can be a model such as Word2Vec and GloVe, such as the Word2Vec and GloVe models in the prior art website "https: / / blog.csdn.net / fs1341825137 / article / details / 121178123", which will not be described in detail in this embodiment.

[0104] The above uses the vector space model to construct the context feature vector of each historical question, calculates the similarity between the context feature vectors corresponding to two historical questions, and obtains the context similarity. Specifically:

[0105] The vector space model is used to construct the context feature vector of each historical question as Y x =(x1,x2,...,x N ), where x N It represents the point mutual information value between each keyword and the context feature word in the nth dimension in each historical question. The point mutual information value is calculated using the point mutual information PMI.

[0106] The above-mentioned point mutual information PMI is a prior art and will not be described in detail in this embodiment.

[0107] The cosine similarity is used to calculate the similarity between the context feature vectors corresponding to the two historical questions:

[0108]

[0109] Among them, sim1(A,B) represents the similarity between the context feature vectors corresponding to the two historical questions, Y x (A) represents the context feature vector corresponding to a historical question, Y x (B) represents the context feature vector corresponding to a historical question.

[0110] The above word vector model is used to construct the semantic feature vector of each historical question, and the similarity between the semantic feature vectors corresponding to two historical questions is calculated to obtain the semantic similarity. Specifically:

[0111] By converting each word in each historical question text into a word vector through the word vector model and calculating the average or weighted average of the text vectors, we can get the vector representation of each historical question text: F x =(f1,f2,...,f N ), where f N Represents the vector representation of the Nth word.

[0112] The cosine similarity is used to calculate the similarity between the semantic feature vectors corresponding to two historical questions:

[0113]

[0114] Among them, sim2(A,B) represents the similarity between the semantic feature vectors corresponding to the two historical questions, and F x (A) represents the semantic feature vector corresponding to a historical question, F x (B) represents the semantic feature vector corresponding to a historical question.

[0115] According to the context similarity and semantic similarity, the fitness function FS is constructed as:

[0116] FS=sim1(A,B)+sim2(A,B)

[0117] The above-mentioned fitness function-based method uses a genetic algorithm to classify each historical question in the acquired historical question set and the document paragraphs related to the historical question set. The above-mentioned fitness function FS can be used as the fitness function in the genetic algorithm, and then the genetic algorithm is used to classify each historical question in the acquired historical question set and the document paragraphs related to the historical question set.

[0118] The above genetic algorithm is a prior art. For example, the genetic algorithm in the prior art website “https: / / cloud.tencent.com / developer / article / 1146073” is used, and this embodiment will not describe it in detail.

[0119] In some implementations, extracting a plurality of first keywords from each type of historical question document set includes:

[0120] A word segmenter is used to segment each historical question in each category of historical question document set to obtain a first word segmentation set;

[0121] Use a word segmenter to segment the document paragraphs related to each historical issue to obtain a second word segmentation set;

[0122] According to the first segmented word set and the second segmented word set, a keyword extraction method is adopted to extract a plurality of first keywords in the first segmented word set.

[0123] In this embodiment, each historical issue in each type of historical issue document set and the document paragraphs related to each historical issue are segmented by a word segmenter, and then a keyword extraction method is used to extract keywords without manual extraction, which can improve the efficiency of keyword extraction.

[0124] In this embodiment, a word segmenter is used to segment each historical issue in each type of historical issue document set to obtain a first word segmentation set; then a word segmenter is used to segment the document paragraphs related to each historical issue to obtain a second word segmentation set; then the frequency of occurrence of each word in the first word segmentation set corresponding to each historical issue in the document paragraph related to each historical issue can be calculated according to the TF-IDF algorithm, and the top few words with the highest occurrence frequency are used as the first keywords.

[0125] The main steps of the TF-IDF algorithm are as follows:

[0126] 1. Word frequency calculation: Calculate the number of times each word appears in the document.

[0127] 2. Inverse document frequency calculation: For each word, calculate the number of documents containing this word, and then take the inverse of this number of documents. This can reduce the IDF value of words that appear in many documents.

[0128] 3. TF-IDF calculation: Multiply the TF and IDF values ​​of each word to get the TF-IDF value of the word. This value represents the importance of the word in the entire document set.

[0129] 4. Keyword extraction: Sort all words according to the TF-IDF value and select the top-ranked words as the keywords of the document.

[0130] In some implementations, constructing a keyword library including a plurality of first keywords and words related to the first keywords includes:

[0131] Combining multiple first keywords in all categories of historical question document sets into a keyword set;

[0132] Find at least one of a synonym and a near synonym for each keyword in the keyword set;

[0133] Each keyword and at least one of a synonym and a near-synonym corresponding to each keyword are combined into a keyword library.

[0134] In this embodiment, a keyword library is constructed that includes multiple first keywords and at least one of synonyms and near-synonyms related to the first keywords. This can reduce the probability of obtaining incorrect answers due to the inability to find the same keywords in the relational knowledge graph for the keywords of the questions asked by the user. Relatively correct answers can be obtained through similar words to improve the accuracy of obtaining the target answer.

[0135] The above search for synonyms or near synonyms of each keyword in the keyword set may be performed manually or by using a tool for searching synonyms or near synonyms in the prior art. It is also possible to search for words associated with each keyword. For example, synonyms of bank include money house and financial institution, and the associated words include deposit, loan, savings, and other related words.

[0136] In some implementations, searching a keyword library for a plurality of third keywords that are close to the plurality of second keywords includes:

[0137] Using a word vector model to vectorize each second keyword to obtain multiple first keyword vectors;

[0138] Using a word vector model to vectorize each keyword in the keyword library to obtain multiple second keyword vectors;

[0139] Calculating the similarity between each first keyword vector and each second keyword vector to obtain multiple similarity values;

[0140] Sorting multiple similarity values ​​to obtain sorted similarity values;

[0141] A preset number of similarity values ​​are selected from the sorted similarity values, and keywords corresponding to the selected preset number of similarity values ​​are used as third keywords.

[0142] In this embodiment, multiple third keywords that are close to multiple second keywords are searched from the keyword library in order to find some answer document paragraphs that are most relevant to the question to be answered. There is no need to search all the document paragraphs to predict the subsequent target answers. This can improve both the accuracy and efficiency of document question and answering.

[0143] The similarity between each first keyword vector and each second keyword vector is calculated to obtain multiple similarity values, and the cosine similarity or Euclidean distance is used to calculate the similarity between each first keyword vector and each second keyword vector.

[0144] In some embodiments, the trained prediction model is obtained by:

[0145] Based on the LLM model, a large prediction model is constructed;

[0146] The prediction model is trained using several historical issues, document paragraphs related to the historical issues, and relational knowledge graphs to obtain a trained prediction model.

[0147] In this embodiment, a prediction big model is constructed based on the LLM model, and the prediction big model is trained by comprehensively considering several historical issues, document paragraphs related to the historical issues, and relational knowledge graphs, which can improve the accuracy of the prediction big model.

[0148] The above-mentioned prediction large model is constructed based on the LLM model. It can be constructed by only using a large language model, or by combining the LLM model with a support vector machine to construct the prediction large model.

[0149] The above-mentioned combination of the LLM model and the support vector machine can be a combination method in the prior art. For example, the LLM model is used for preliminary prediction in the early stage, and the support vector machine is further used to classify the results of the preliminary prediction in the later stage, thereby improving the accuracy of the prediction model. The LLM model and the support vector machine are both technical solutions well known to those skilled in the art, and are not described in detail in this embodiment.

[0150] The above-mentioned relational knowledge graph can be implicitly added to the training of the prediction model, and the injection can be completed by embedding. The implementation of KnowBERT can be referred to. The method of training the prediction model with the relational knowledge graph adopted in this embodiment is the same as that in the prior art, and will not be described in detail in this embodiment.

[0151] In some embodiments, after inputting the question to be answered and multiple document paragraphs into the trained prediction model to obtain the target answer to the question to be answered, the document question answering method based on LLM, RAG and knowledge graph further includes:

[0152] Input the question to be answered and multiple document paragraphs into the trained prediction model to predict the target answer to the question to be answered;

[0153] The target answer and the unanswered question corresponding to the target answer are recorded as historical records;

[0154] The historical record is used to update the historical question collection and the document sections related to the historical question collection.

[0155] In this embodiment, historical records are used to update the historical question set and document paragraphs related to the historical question set, which can better find the correct target answers for later predictions of the same type of unanswered questions, and the trained large prediction model can directly obtain the target answers to the unanswered questions, thereby improving the efficiency of finding the target answers.

[0156] Reference Figure 2 The embodiment of the present application also provides a document question-answering system based on LLM, RAG and knowledge graph. The system includes a data classification unit 100, a keyword extraction unit 200, a knowledge graph construction unit 300, a keyword library construction unit 400, a keyword search unit 500, a document paragraph retrieval unit 600 and a target answer prediction unit 700, wherein:

[0157] The data classification unit 100 is used to classify the acquired historical question set and the document paragraphs related to the historical question set to obtain multiple categories of historical question document sets, wherein one historical question document set includes a plurality of historical questions of the same type and document paragraphs related to the plurality of historical questions;

[0158] A keyword extraction unit 200 is used to extract a plurality of first keywords from each type of historical problem document set, where the plurality of first keywords are keywords selected from the historical problem document set;

[0159] The knowledge graph construction unit 300 is used to construct the relationship between the first keywords and the document paragraphs related to the first keywords by using the knowledge graph, so as to obtain the relationship knowledge graph corresponding to each type of historical problem document set;

[0160] A keyword library construction unit 400 is used to construct a keyword library including a plurality of first keywords and words related to the first keywords;

[0161] The keyword search unit 500 is used to extract multiple second keywords from the question to be answered, and search for multiple third keywords similar to the multiple second keywords from the keyword library, where the second keywords are keywords extracted from the question to be answered, and the third keywords are keywords searched from the keyword library;

[0162] A document paragraph retrieval unit 600 is used to retrieve multiple document paragraphs corresponding to multiple second keywords and multiple third keywords from the relational knowledge graph using the RAG method;

[0163] The target answer prediction unit 700 is used to input the question to be answered and multiple document paragraphs into the trained prediction model to obtain the target answer to the question to be answered.

[0164] It should be noted that since the document question and answer system based on LLM, RAG and knowledge graph in this embodiment and the above-mentioned document question and answer method based on LLM, RAG and knowledge graph are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0165] Reference Figure 3The embodiment of the present application further provides an electronic device, the electronic device comprising:

[0166] at least one memory;

[0167] at least one processor;

[0168] at least one program;

[0169] The program is stored in the memory, and the processor executes at least one program to implement the document question and answer method based on LLM, RAG and knowledge graph implemented in the present disclosure.

[0170] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0171] The electronic device according to the embodiment of the present application is described in detail below.

[0172] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure;

[0173] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1700, and the processor 1600 calls and executes the document question-and-answer method based on LLM, RAG, and knowledge graph in the embodiment of the present disclosure.

[0174] Input / output interface 1800, used to implement information input and output;

[0175] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0176] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0177] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0178] An embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned document question and answer method based on LLM, RAG and knowledge graph.

[0179] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0180] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0181] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present disclosure and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0182] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0184] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0185] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0186] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0189] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs. The above is a detailed description of the embodiments of the present application in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Within the scope of knowledge possessed by ordinary technicians in the relevant technical field, various changes can be made without departing from the purpose of the present application.

Claims

1. A document question answering method based on LLM, RAG and knowledge graph, characterized in that: The method comprises: Classifying the acquired historical question set and document paragraphs related to the historical question set to obtain multiple categories of historical question document sets, wherein one historical question document set includes a plurality of historical questions of the same type and document paragraphs related to the plurality of historical questions; Extracting a plurality of first keywords from each category of the historical problem document set, wherein the plurality of first keywords are keywords selected from the historical problem document set; Using the knowledge graph to construct the relationship between the multiple first keywords and the document paragraphs related to the multiple first keywords, to obtain a relationship knowledge graph corresponding to each type of the historical question document set; constructing a keyword library including the plurality of first keywords and words related to the first keywords; Extracting multiple second keywords from the question to be answered, and searching the keyword library for multiple third keywords that are close to the multiple second keywords, wherein the second keywords are keywords extracted from the question to be answered, and the third keywords are keywords searched from the keyword library; Retrieving a plurality of document paragraphs corresponding to the plurality of second keywords and the plurality of third keywords from the relational knowledge graph using a RAG method; The question to be answered and the multiple document paragraphs are input into the trained prediction model to obtain the target answer to the question to be answered.

2. The document question-answering method based on LLM, RAG and knowledge graph according to claim 1, characterized in that: The acquired historical question set and document paragraphs related to the historical question set are classified to obtain multiple categories of historical question document sets, including: A vector space model is used to construct a context feature vector for each of the historical issues; Calculating the similarity between the context feature vectors corresponding to the two historical questions to obtain context similarity; Using a word vector model to construct a semantic feature vector for each of the historical questions; Calculating the similarity between the semantic feature vectors corresponding to the two historical questions to obtain semantic similarity; Constructing a fitness function according to the context similarity and the semantic similarity; Based on the fitness function, a genetic algorithm is used to classify each historical question in the acquired historical question set and document paragraphs related to the historical question set to obtain a multi-category historical question document set.

3. The document question-answering method based on LLM, RAG and knowledge graph according to claim 1, characterized in that: The step of extracting a plurality of first keywords from each type of the historical problem document set includes: Using a word segmenter to segment each historical issue in each category of the historical issue document set to obtain a first word segmentation set; Using a word segmenter to segment the document paragraphs related to each of the historical issues to obtain a second word segmentation set; According to the first segmented word set and the second segmented word set, a keyword extraction method is adopted to extract a plurality of first keywords in the first segmented word set.

4. The document question-answering method based on LLM, RAG and knowledge graph according to claim 1, characterized in that: The step of constructing a keyword library including the plurality of first keywords and words related to the first keywords includes: Combining a plurality of first keywords in the historical question document set of all categories into a keyword set; Find at least one of a synonym and a near synonym for each keyword in the keyword set; Each keyword and at least one of a synonym and a near-synonym corresponding to each keyword are combined into a keyword library.

5. The document question-answering method based on LLM, RAG and knowledge graph according to claim 1, characterized in that: The step of searching the keyword library for a plurality of third keywords that are close to the plurality of second keywords includes: Using a word vector model to vectorize each of the second keywords to obtain multiple first keyword vectors; Using a word vector model to vectorize each keyword in the keyword library to obtain multiple second keyword vectors; Calculating the similarity between each of the first keyword vectors and each of the second keyword vectors to obtain multiple similarity values; Sorting the multiple similarity values ​​to obtain sorted similarity values; A preset number of similarity values ​​are selected from the sorted similarity values, and keywords corresponding to the selected preset number of similarity values ​​are used as third keywords.

6. The document question-answering method based on LLM, RAG and knowledge graph according to claim 1, characterized in that: The trained prediction model is obtained in the following way: Based on the LLM model, a large prediction model is constructed; The prediction big model is trained using the acquired several historical issues, document paragraphs related to the historical issues, and the relational knowledge graph to obtain a trained prediction big model.

7. The document question-answering method based on LLM, RAG and knowledge graph according to claim 1, characterized in that: After inputting the question to be answered and the multiple document paragraphs into the trained prediction model to obtain the target answer to the question to be answered, the document question answering method based on LLM, RAG and knowledge graph further includes: Inputting the question to be answered and the multiple document paragraphs into the trained prediction model to predict the target answer to the question to be answered; Recording the target answer and the unanswered question corresponding to the target answer as historical records; The historical record is used to update the historical question set and the document paragraphs related to the historical question set.

8. A document question answering system based on LLM, RAG and knowledge graph, characterized in that: The system comprises: A data classification unit is used to classify the acquired historical question set and the document paragraphs related to the historical question set to obtain multiple categories of historical question document sets, wherein one historical question document set includes a plurality of historical questions of the same type and document paragraphs related to the plurality of historical questions; A keyword extraction unit, used to extract a plurality of first keywords from each type of the historical problem document set, wherein the plurality of first keywords are keywords selected from the historical problem document set; A knowledge graph construction unit, configured to use the knowledge graph to construct the relationship between the plurality of first keywords and the document paragraphs related to the plurality of first keywords, and obtain a relationship knowledge graph corresponding to each type of the historical question document set; A keyword library construction unit, used to construct a keyword library including the plurality of first keywords and words related to the first keywords; A keyword search unit, used to extract a plurality of second keywords from the question to be answered, and search for a plurality of third keywords close to the plurality of second keywords from the keyword library, wherein the second keywords are keywords extracted from the question to be answered, and the third keywords are keywords searched from the keyword library; A document paragraph retrieval unit, configured to retrieve a plurality of document paragraphs corresponding to the plurality of second keywords and the plurality of third keywords from the relational knowledge graph using a RAG method; The target answer prediction unit is used to input the question to be answered and the multiple document paragraphs into the trained prediction model to obtain the target answer to the question to be answered.

9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the document question and answer method based on LLM, RAG and knowledge graph as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the document question and answer method based on LLM, RAG and knowledge graph as described in any one of claims 1 to 7.

Images