Policy Q&A System Based on Large Language Model, Inverted Index, and Embedding Retrieval

Abstract

The present invention discloses a policy Q&A system based on large language models, inverted indexing, and embedding retrieval, including: a data preprocessing module, an inverted indexing module, an embedding retrieval module, a retrieval management module, a Q&A generation module, and a user interface module; relating to the technical field of policy Q&A, it solves the technical problem that existing policy Q&A systems often cannot balance retrieval efficiency and semantic understanding at the same time, resulting in insufficient accuracy and relevance of answers; through a two-stage retrieval mechanism that combines inverted indexing and embedding retrieval, first, candidate policy texts are quickly screened out through inverted indexing, and then several policy texts and the demand questions are transformed into two high-dimensional vectors and compared. According to the comparison results, fine sorting is carried out, and the large language model answers with reference to several candidate policy texts that have been finely sorted, which can improve the relevance and accuracy of retrieval results.

Description

Technical Field

[0001] The present invention belongs to the technical field of policy Q&A, and specifically relates to a policy Q&A system based on large language models, inverted indexing, and embedding retrieval. Background Art

[0002] In the fields of information retrieval and Q&A systems, traditional inverted indexing methods and embedding-based retrieval methods each have their own advantages and disadvantages. The inverted indexing method is fast and efficient, but has weak semantic understanding; the embedding retrieval method has strong semantic understanding ability, but has high computational complexity. Large language models, due to their powerful generation and semantic understanding capabilities, mainly predict answers based on questions, and there are problems such as talking nonsense and making random associations, and do not have a retrieval function. Existing policy Q&A systems often cannot balance retrieval efficiency and semantic understanding at the same time, resulting in insufficient accuracy and relevance of answers. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art; for this purpose, the present invention proposes a policy Q&A system based on large language models, inverted indexing, and embedding retrieval, which is used to solve the technical problem that existing policy Q&A systems often cannot balance retrieval efficiency and semantic understanding at the same time, resulting in insufficient accuracy and relevance of answers. The present invention quickly screens out candidate policy texts through inverted indexing, then converts several policy texts and demand questions into two high-dimensional vectors and compares them, and performs fine sorting according to the comparison results. The large language model answers based on the fine-sorted several candidate policy texts to solve the above problems.

[0004] To achieve the above object, the first aspect of the present invention provides a policy Q&A system based on large language models, inverted indexing, and embedding retrieval, including: a data preprocessing module, an inverted indexing module, an embedding retrieval module, a retrieval management module, a Q&A generation module, and a user interface module;

[0005] The data preprocessing module: used to obtain policy texts and demand questions; and clean the texts in the policy texts; wherein, the cleaning methods include word segmentation, stop word removal, and stemming extraction;

[0006] The inverted indexing module: creates an inverted index based on the cleaned policy texts, and constructs a term-document inverted list based on the text index results; and after performing inverted indexing on the demand questions, matches them with the term-document inverted list to obtain several candidate policy texts;

[0007] The embedding retrieval module: converts the demand questions into question high-dimensional vectors through a word embedding module; converts the cleaned policy texts into policy high-dimensional vectors through a word embedding model, and constructs a policy vector index;

[0008] The retrieval management module: matches the high-dimensional vector of the question with a number of high-dimensional vectors of policies to obtain a matching index; wherein, the matching index consists of a number of candidate policy texts with a sequence;

[0009] The question-answering generation module: the large language model generates an answer corresponding to the demand question according to the matching index; wherein, the large language model is established based on an artificial intelligence model;

[0010] The user interface module: is used to input the demand question; and, display the answer corresponding to the demand question and the corresponding policy text.

[0011] It should be noted that: the policy vector index is used to index the position of the high-dimensional vector of the policy in the policy text, and when the question-answering generation module generates an answer, the large model generates it according to the sequence of the policy texts in the matching index and the policy vector index.

[0012] Preferably, the large language model is established based on an artificial intelligence model, including:

[0013] Obtain a quality fine-tuning data set and a training data set, and build a large language model to be trained through the llama3 model and add dynamic weight adjustment to the output layer of the llama3 model and introduce a self-attention mechanism;

[0014] Perform low-rank decomposition on the weight matrix in the large language model to be trained; through low-rank decomposition, the weight matrix W can be decomposed into two smaller matrices A and B, and W is approximately equal to A×B; where the ranks of matrices A and B are much smaller than the dimension of the original weight matrix, and this kind of decomposition greatly reduces the number of parameters to be trained;

[0015] Insert a low-rank adapter into a specific layer of the model; wherein, the specific layer is any dense layer of the model, including the weight matrix of the self-attention layer;

[0016] Train the large language model to be trained through the training data set, and test the trained large language model through the quality fine-tuning data set, and adjust the trained large language model according to the test results to finally obtain the trained large language model.

[0017] It should be noted that this large language model can output answers without dynamic weight adjustment and with dynamic weight adjustment.

[0018] Preferably, constructing the term-document inverted list based on the text index result includes:

[0019] Obtain the text index result of the reverse index of the cleaned policy text, count the number of occurrences of several terms in the text index result, and obtain the document ID of the policy text where the corresponding term is located and the position information in the corresponding policy text; draw several terms, the document ID of the policy text where the term is located, and the position in the corresponding policy text into a permutation table to obtain a term-document inverted list.

[0020] Preferably, after performing a reverse index on the demand problem and matching it with the term-document inverted list, it includes:

[0021] Clean the demand problem, and perform a reverse index on the cleaned demand problem to obtain a problem index result;

[0022] Match the terms in the problem index result with the terms in the term-document inverted list to obtain matching terms, and mark the several policy text document IDs recorded by the matching terms in the term-document inverted list as screened policy texts;

[0023] Screen out the policy text with the highest relevance from the screened policy texts and mark it as the candidate policy text.

[0024] Preferably, the matching of the problem high-dimensional vector and several policy high-dimensional vectors includes:

[0025] Calculate the Euclidean distances between the problem high-dimensional and several policy high-dimensional vectors respectively, and mark them as Li; where i = 1, 2,... n, and n is the number of policy high-dimensional vectors;

[0026] Sort the policy texts corresponding to several Euclidean distances Li from largest to smallest to obtain a matching index.

[0027] Preferably, the system further includes an implementation optimization module, and the user interface module: click to view the candidate policy texts in the matching index and give feedback on the answers corresponding to the demand problems; where the feedback results include satisfied and dissatisfied;

[0028] The implementation optimization module: used to obtain the feedback results of the user on the answers corresponding to several demand problems; adjust the order of several candidate policy texts in the matching index based on several feedback results.

[0029] Preferably, the adjustment of the order of several candidate policy texts in the matching index based on several feedback results includes:

[0030] Obtain the feedback results corresponding to the answers that have not undergone dynamic weight adjustment and those that have undergone dynamic weight adjustment among several demand problems; assign satisfied and dissatisfied in the feedback results as a and b respectively, and a is greater than b;

[0031] Sum the assignments without dynamic weight adjustment and the assignments with dynamic weight adjustment respectively to obtain the unwrapped value and the wrapped value;

[0032] Determine whether the difference between the unwrapped value and the wrapped value is less than the difference threshold; if yes, keep the order of the candidate policy texts; if no, sort the policy text at the top of the matching index to the last position.

[0033] Compared with the prior art, the beneficial effects of the present invention are:

[0034] 1. In the present invention, through a two-stage retrieval mechanism that combines reverse indexing and embedding retrieval, first quickly screen out candidate policy texts through reverse indexing, and then convert several policy texts and demand questions into two high-dimensional vectors and compare them. According to the comparison results, perform fine sorting, and answer with reference to the several candidate policy texts sorted by the large language model in a refined manner, which can improve the relevance and accuracy of the retrieval results.

[0035] 2. In the present invention, add dynamic weight adjustment to the output layer of the llama3 model, and perform context connection and optimize the matching index according to multiple demand questions proposed by the user, so as to make the subsequent demand questions more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0037] Figure 1 It is a schematic diagram of the connection of modules in the present invention;

[0038] Figure 2 It is a schematic diagram of the working principle of the policy question-answering system based on the large language model, reverse indexing and embedding retrieval in the present invention; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0040] Please refer to Figure 1 - Figure 2, an embodiment of the first aspect of the present invention provides a policy Q&A system based on large language models, inverted indexing, and embedding retrieval, including: a data preprocessing module, an inverted indexing module, an embedding retrieval module, a retrieval management module, a Q&A generation module, and a user interface module;

[0041] Please refer to Figure 1 As shown, the data preprocessing module is communicatively connected to the inverted indexing module, the inverted indexing module is communicatively connected to the embedding retrieval module, both the inverted indexing module and the embedding retrieval module are communicatively connected to the retrieval management module, the retrieval management module is communicatively connected to the Q&A generation module, and the Q&A generation module is communicatively connected to the user interface module;

[0042] Data preprocessing module: used to obtain policy texts and demand questions; and, clean the texts in the policy texts; wherein, the cleaning methods include word segmentation, stop word removal, and stemming;

[0043] Inverted indexing module: create an inverted index based on the cleaned policy texts, and construct a term-document inverted list based on the text indexing results; and, match the demand questions after inverted indexing with the term-document inverted list to obtain several candidate policy texts;

[0044] Embedding retrieval module: convert the demand questions into question high-dimensional vectors through the word embedding module; convert the cleaned policy texts into policy high-dimensional vectors through the word embedding model, and construct a policy vector index;

[0045] Retrieval management module: match the question high-dimensional vectors with several policy high-dimensional vectors to obtain a matching index; wherein, the matching index consists of several candidate policy texts with a sequential order;

[0046] Q&A generation module: the large language model generates an answer corresponding to the demand question according to the matching index; wherein, the large language model is established based on an artificial intelligence model;

[0047] User interface module: used to input demand questions; and, display the answers corresponding to the demand questions and the corresponding policy texts.

[0048] The above-mentioned term-document inverted list is a mapping of the terms (words or phrases) in a document to a list of documents containing these terms, so as to quickly find all documents containing a certain term. It is arranged according to the number of occurrences of the term. In the inverted list, record the documents and their positions where each term appears to achieve fast retrieval.

[0049] Specifically, the large language model is established based on an artificial intelligence model, including:

[0050] Obtain the quality fine-tuning dataset and the training dataset, and construct a large language model to be trained through the llama3-chinese-chat-7B model, adding dynamic weight adjustment to the output layer of the llama3-chinese-chat-7B model and introducing the self-attention mechanism.

[0051] Perform low-rank decomposition on the weight matrix in the large language model to be trained; through low-rank decomposition, the weight matrix W can be decomposed into two smaller matrices A and B, and W is approximately equal to A×B; where the ranks of matrices A and B are much smaller than the dimension of the original weight matrix, and this decomposition greatly reduces the number of parameters to be trained.

[0052] Insert a low-rank adapter into the weight matrix of the self-attention layer of the model.

[0053] Train the large language model to be trained with the training dataset, and test the trained large language model with the quality fine-tuning dataset. Adjust the trained large language model according to the test results, and finally obtain a large language model with the input being the demand question and the output being the answers without dynamic weight adjustment and with dynamic weight adjustment.

[0054] Furthermore, construct a term-document inverted list based on the text index results, including:

[0055] Obtain the text index results of the inverted index of the cleaned policy text, count the number of occurrences of several terms in the text index results, and obtain the document ID of the policy text where the corresponding term is located and the position information in the corresponding policy text; draw several terms, the document ID of the policy text where the corresponding term is located, and the position in the corresponding policy text into the inverted list to obtain the term-document inverted list.

[0056] Exemplarily, there are the following policy texts:

[0057] Doc1: "The maternity leave for women in Beijing is 128 days";

[0058] Doc2: "The maternity leave for women in Shanghai is 128 days";

[0059] Doc3: "The maternity leave for women in Guangdong Province is 158 days";

[0060] Doc4: "The maternity leave for women in Inner Mongolia Autonomous Region is 158 days";

[0061] Doc5: "The maternity leave for women in Guangxi Zhuang Autonomous Region is 190 days";

[0062] Doc6: "The maternity leave for women in Ningxia Hui Autonomous Region is 180 days";

[0063] Doc7: "The female care leave in Ningxia Hui Autonomous Region is 15 days";

[0064] Doc8: "The male care leave in Guangxi Zhuang Autonomous Region is 18 days";

[0065] Then the constructed term-document inverted list is:

[0066] Beijing: [Doc1]

[0067] Shanghai: [Doc2]

[0068] Guangdong Province: [Doc3]

[0069] Inner Mongolia Autonomous Region: [Doc4]

[0070] Guangxi Zhuang Autonomous Region: [Doc5, Doc8]

[0071] Ningxia Hui Autonomous Region: [Doc6, Doc7]

[0072] Female: [Doc1, Doc2, Doc3, Doc4, Doc5, Doc6, Doc7]

[0073] Maternity leave: [Doc1, Doc2, Doc3, Doc4, Doc5, Doc6]

[0074] Care leave: [Doc7, Doc8]

[0075] Male: [Doc8]

[0076] 128 days: [Doc1, Doc2]

[0077] 158 days: [Doc3, Doc4]

[0078] 190 days: [Doc5]

[0079] 180 days: [Doc6]

[0080] 15 days: [Doc7]

[0081] 18 days: [Doc8];

[0082] It should be noted that for the sake of simplicity, this example only gives a simple example of policy texts, so the positions in the corresponding policy texts are omitted. The following exemplary content also omits the same content and will not be elaborated here.

[0083] After reverse indexing the demand questions and matching them with the term-document inverted list, it includes:

[0084] Clean the demand questions and perform reverse indexing on the cleaned demand questions to obtain the question indexing results;

[0085] Match the terms in the problem index result with the terms in the term-document inverted list to obtain matching terms, and mark the several policy document IDs recorded by the matching terms in the term-document inverted list as screened policy texts;

[0086] Screen out the policy text with the highest relevance from the screened policy texts and mark it as the candidate policy text.

[0087] Exemplarily, if the requirement problem is: How many days is the maternity leave for women in Ningxia Hui Autonomous Region?, then the content after cleaning and indexing this problem is: [Ningxia Hui Autonomous Region, women, maternity leave]; and the matching terms are [Ningxia Hui Autonomous Region, women, maternity leave], and the policy texts corresponding to the matching terms are:

[0088] Ningxia Hui Autonomous Region: [Doc6, Doc7];

[0089] Women: [Doc1, Doc2, Doc3, Doc4, Doc5, Doc6, Doc7];

[0090] Maternity leave: [Doc1, Doc2, Doc3, Doc4, Doc5, Doc6];

[0091] It can be known that the screened texts are Doc1, Doc2, Doc3, Doc4, Doc5, Doc6, Doc7;

[0092] And Doc6 has the highest relevance, so Doc6 is the candidate policy text.

[0093] Furthermore, match the problem high-dimensional vector with several policy high-dimensional vectors, including:

[0094] Calculate the Euclidean distances between the problem high-dimensional and several policy high-dimensional vectors respectively, and mark them as Li; where, i = 1, 2,... n, and n is the number of policy high-dimensional vectors;

[0095] Sort the policy texts corresponding to several Euclidean distances Li from largest to smallest to obtain the matching index.

[0096] It should be noted that since there is only one candidate policy text, Doc6, in the above exemplary content, therefore, in this example, this step can be skipped. If there are multiple candidate policy texts, the problem high-dimensional vector and several policy high-dimensional vectors can be matched.

[0097] In another real-time example, a policy Q&A system based on a large language model, reverse indexing, and embedding retrieval further includes an implementation optimization module; the real-time optimization module is communicatively connected to the Q&A management module.

[0098] The user interface module in this embodiment: clicks to view the candidate policy texts in the matching index and gives feedback on the answers corresponding to the requirement questions; the feedback results include satisfied and dissatisfied;

[0099] Specifically, the optimization module: is used to obtain the feedback results of the user on the answers corresponding to several requirement questions; adjusts the order of several candidate policy texts in the matching index based on the several feedback results.

[0100] By adjusting the order of the candidate policy texts, the accuracy of the answers to the requirement questions proposed subsequently can be further improved.

[0101] Furthermore, adjusting the order of several candidate policy texts in the matching index based on several feedback results includes:

[0102] Obtain the feedback results corresponding to the answers that have not undergone dynamic weight adjustment and those that have undergone dynamic weight adjustment among several requirement questions; assign 1 and 0 to satisfied and dissatisfied respectively in the feedback results

[0103] Sum up the assignments that have not undergone dynamic weight adjustment and those that have undergone dynamic weight adjustment respectively to obtain the unencapsulated value and the encapsulated value;

[0104] Judge whether the difference between the unencapsulated value and the encapsulated value is less than the difference threshold 1; if yes, keep the order of the candidate policy texts; if not, sort the policy text at the top of the matching index to the last position.

[0105] The working principle of the present invention: The user inputs requirement questions through the user interface module, and the reverse indexing module performs reverse indexing on the requirement questions to obtain several candidate policy texts;

[0106] The candidate policy texts are transformed into several policy high-dimensional vectors through the embedded retrieval module, and the requirement questions are transformed into question high-dimensional vectors through the word embedding module;

[0107] Calculate the similarity between the question high-dimensional vector and several policy high-dimensional vectors respectively to obtain several similarity values, sort the several candidate policy texts corresponding to the several similarity values in descending order to obtain a sorted list; the large language model generates the answers corresponding to the requirement questions according to the policy text ranked first in the sorted list.

[0108] It should be noted that obtaining the policy text; cleaning the text in the policy text; creating a reverse index based on the cleaned policy text and constructing a term-document inverted list based on the text index result; transforming the cleaned policy text into a policy high-dimensional vector through a word embedding model and constructing a policy vector index; these steps have been completed before the user inputs the requirement questions.

[0109] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical method of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A policy Q&A system based on large language models, inverted indexing, and embedding retrieval, characterized in that, Including: A data preprocessing module, an inverted index module, an embedding retrieval module, a retrieval management module, a question-and-answer generation module, an implementation optimization module, and a user interface module; The data preprocessing module: used to obtain policy texts and demand questions; and, clean the texts in the policy texts; wherein, the cleaning methods include word segmentation, stop word removal, and stemming; The inverted index module: create an inverted index based on the cleaned policy texts, and construct a term-document inverted list based on the text index results; and, match the demand questions after inverted indexing with the term-document inverted list to obtain several candidate policy texts; The embedding retrieval module: convert the demand questions into high-dimensional question vectors through a word embedding module; convert the cleaned policy texts into high-dimensional policy vectors through a word embedding model; The retrieval management module: match the high-dimensional question vectors and several high-dimensional policy vectors to obtain a matching index; wherein, the matching index consists of several candidate policy texts with a sequential order; The question-and-answer generation module: the large language model generates answers corresponding to the demand questions according to the matching index; wherein, the large language model is established based on an artificial intelligence model; The user interface module: used to input demand questions; and, display the answers corresponding to the demand questions and the corresponding policy texts; The implementation optimization module: used to obtain the feedback results of the user on the answers corresponding to several demand questions; adjust the order of several candidate policy texts in the matching index based on several feedback results, wherein the feedback results include satisfied and dissatisfied; Adjusting the order of several candidate policy texts in the matching index based on several feedback results includes: Obtain the feedback results corresponding to the answers that have not undergone dynamic weight adjustment and those that have undergone dynamic weight adjustment among several demand questions; assign satisfied and dissatisfied in the feedback results as a and b respectively, and a is greater than b; Sum up the assignments of those that have not undergone dynamic weight adjustment and those that have undergone dynamic weight adjustment respectively to obtain an unencapsulated value and an encapsulated value; Judge whether the difference between the unencapsulated value and the encapsulated value is less than the difference threshold; if so, keep the order of the candidate policy texts; otherwise, sort the policy text at the top of the matching index to the last position.

2. The policy Q&A system based on large language models, inverted indexing, and embedding retrieval according to claim 1, wherein The large language model is established based on an artificial intelligence model, including: Obtain a quality fine-tuning data set and a training data set, and build a large language model to be trained through the llama3 model and add dynamic weight adjustment to the output layer of the llama3 model and introduce a self-attention mechanism; Perform low-rank decomposition on the weight matrix in the large language model to be trained; Insert a low-rank adapter into a specific layer of the model; wherein, the specific layer is any dense layer of the model, including the weight matrix of the self-attention layer; Train the large language model to be trained through the training data set, and test the trained large language model through the quality fine-tuning data set, and adjust the trained large language model according to the test results to finally obtain the trained large language model.

3. The policy Q&A system based on large language models, inverted indexing, and embedding retrieval according to claim 1, wherein Constructing a term-document inverted list based on the text index results includes: Obtain the text index results of the reverse index of the cleaned policy text, count the number of occurrences of several terms in the text index results, and obtain the document IDs of the policy texts where the corresponding terms are located and the position information in the corresponding policy texts; draw several terms, the document IDs of the policy texts where the corresponding terms are located, and the positions in the corresponding policy texts into a permutation table to obtain a term-document inverted permutation table.

4. The policy Q&A system based on large language models, inverted indexing, and embedding retrieval according to claim 1, wherein The matching after reverse indexing the demand problem with the term-document inverted permutation table includes: Clean the demand problem, and perform reverse indexing on the cleaned demand problem to obtain problem index results; Match the terms in the problem index results with the terms in the term-document inverted permutation table to obtain matching terms, and mark the several policy text document IDs recorded by the matching terms in the term-document inverted permutation table as screened policy texts; Screen out the policy text with the highest relevance from the screened policy texts and mark it as the candidate policy text.

5. The policy Q&A system based on large language models, inverted indexing, and embedding retrieval according to claim 1, wherein The matching of the problem high-dimensional vector and several policy high-dimensional vectors includes: Calculate the Euclidean distances between the problem high dimension and several policy high-dimensional vectors respectively, and mark them as Li; where i = 1, 2,... n, and n is the number of policy high-dimensional vectors; Sort the policy texts corresponding to several Euclidean distances Li in descending order to obtain a matching index.

6. The policy Q&A system based on large language models, inverted indexing, and embedding retrieval according to claim 1, wherein The user interface module: click to view the candidate policy texts in the matching index and give feedback on the answers corresponding to the demand problems.

Images