Policy question-answering system based on large language model, reverse index and embedded retrieval

Abstract

The invention discloses a policy question and answer system based on a large language model, a reverse index and embedded retrieval. The policy question and answer system comprises a data preprocessing module, a reverse index module, an embedded retrieval module, a retrieval management module, a question and answer generation module and a user interface module. Relates to the technical field of policy questioning and answering, and solves the technical problem that an existing policy questioning and answering system often cannot give consideration to retrieval efficiency and semantic understanding at the same time, so that the accuracy and correlation of answers are insufficient. Through a two-stage retrieval mechanism combining reverse indexing and embedded retrieval, firstly, candidate policy texts are quickly screened out through reverse indexing, then, a plurality of policy texts and demand problems are converted into two high-dimensional vectors and are compared, fine sorting is performed according to comparison results, and the policy texts and demand problems are accurately sorted. And the candidate policy texts which are finely sorted by the large language model are taken as references for answering, so that the correlation and accuracy of retrieval results can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of policy question answering, and in particular is a policy question answering system based on a large language model, reverse indexing and embedded retrieval. Background Art

[0002] In the field of information retrieval and question-answering systems, traditional reverse indexing methods and embedding-based retrieval methods each have their own advantages and disadvantages. The reverse indexing method is fast and efficient, but has weak semantic understanding; the embedding retrieval method has strong semantic understanding capabilities, but has high computational complexity. Large language models have powerful generation and semantic understanding capabilities, but they mainly predict answers based on questions, and have problems such as nonsense and random connections, and do not have retrieval functions. Existing policy question-answering systems often cannot take into account both retrieval efficiency and semantic understanding at the same time, resulting in insufficient accuracy and relevance of answers. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a policy question-answering system based on a large language model, inverted indexing and embedded retrieval, which is used to solve the technical problem that the existing policy question-answering system often cannot take into account both retrieval efficiency and semantic understanding at the same time, resulting in insufficient accuracy and relevance of the answers. The present invention quickly screens out candidate policy texts through a reverse index, and then converts several policy texts and demand questions into two high-dimensional vectors and compares them, performs fine sorting according to the comparison results, and uses several candidate policy texts finely sorted by the large language model as a reference for answering to solve the above-mentioned problem.

[0004] To achieve the above-mentioned object, the first aspect of the present invention provides a policy question-answering system based on a large language model, a reverse index and an embedded retrieval, comprising: a data preprocessing module, a reverse index module, an embedded retrieval module, a retrieval management module, a question-answering generation module and a user interface module;

[0005] The data preprocessing module is used to obtain policy texts and demand issues; and to clean the text in the policy text; wherein the cleaning methods include word segmentation, stop word removal and stem extraction;

[0006] The reverse index module: creates a reverse index based on the cleaned policy text, and builds a term-document inverted list based on the text index result; and, after reverse indexing the demand question, matches it with the term-document inverted list to obtain a number of candidate policy texts;

[0007] The embedding retrieval module: converts the demand question into a high-dimensional question vector through the word embedding module; converts the cleaned policy text into a high-dimensional policy vector through the word embedding model, and constructs a policy vector index;

[0008] The retrieval management module: matches the problem high-dimensional vector with a plurality of policy high-dimensional vectors to obtain a matching index; wherein the matching index is composed of a plurality of candidate policy texts with a sequence;

[0009] The question-answer generation module: the large language model generates answers corresponding to the required questions according to the matching index; wherein the large language model is established based on the artificial intelligence model;

[0010] The user interface module is used for inputting requirement questions and displaying answers to the requirement questions and corresponding policy texts.

[0011] It should be noted that the policy vector index is used to index the position of the policy high-dimensional vector in the policy text, and when the question-answer generation module generates answers, the large model is generated according to the order of the policy text in the matching index and the policy vector index.

[0012] Preferably, the large language model is established based on an artificial intelligence model, including:

[0013] Obtain quality fine-tuning datasets and training datasets, build a large language model to be trained through the llama3 model, add dynamic weight adjustment to the output layer of the llama3 model, and introduce the self-attention mechanism;

[0014] The weight matrix in the large language model to be trained is decomposed into low rank. The weight matrix W can be decomposed into two smaller matrices A and B by low rank decomposition, and W is approximately equal to A×B. The ranks of matrices A and B are much smaller than the dimensions of the original weight matrix. This decomposition greatly reduces the number of parameters that need to be trained.

[0015] Insert a low-rank adapter at a specific layer of the model; the specific layer is any dense layer of the model, including the weight matrix of the self-attention layer;

[0016] The large language model to be trained is trained through the training data set, and the trained large language model is tested through the quality fine-tuning data set. The trained large language model is adjusted according to the test results, and finally the trained large language model is obtained.

[0017] It should be noted that the large language model can output answers without dynamic weight adjustment as well as answers after dynamic weight adjustment.

[0018] Preferably, the step of constructing a term-document inverted list based on the text index result includes:

[0019] Obtain the text index result of the reverse index of the cleaned policy text, count the number of times several terms appear in the text index result, and obtain the document ID of the policy text where the corresponding term is located and the position information in the corresponding policy text; draw several terms, the document ID of the policy text where the term corresponds to, and the position in the corresponding policy text into an arrangement table to obtain a term-document inverted list.

[0020] Preferably, the reverse indexing of the requirement question and then matching it with the term-document inverted list comprises:

[0021] Clean the demand questions, and reverse index the cleaned demand questions to obtain the question index results;

[0022] Matching the terms in the question index result with the terms in the term-document inverted list to obtain matching terms, and marking several policy text document IDs recorded by the matching terms in the term-document inverted list as screening policy texts;

[0023] The policy texts with the highest relevance are selected from the screened policy texts and marked as candidate policy texts.

[0024] Preferably, matching the problem high-dimensional vector with a plurality of policy high-dimensional vectors includes:

[0025] Calculate the Euclidean distance between the problem high-dimensional vector and several policy high-dimensional vectors respectively, and mark them as Li; where i = 1, 2, ... n, n is the number of policy high-dimensional vectors;

[0026] The policy texts corresponding to the Euclidean distances Li are sorted from large to small to obtain a matching index.

[0027] Preferably, the system further comprises an implementation optimization module, wherein the user interface module: clicks to view the candidate policy texts in the matching index and provides feedback on the answers corresponding to the demand questions; wherein the feedback results include satisfaction and dissatisfaction;

[0028] The implementation optimization module is used to obtain feedback results of users' answers corresponding to a number of demand questions; and to adjust the order of a number of candidate policy texts in the matching index based on the feedback results.

[0029] Preferably, adjusting the order of several candidate policy texts in the matching index based on several feedback results includes:

[0030] Obtain the feedback results corresponding to the answers to several demand questions that have not been dynamically weighted and those that have been dynamically weighted; assign the satisfaction and dissatisfaction in the feedback results to a and b respectively, and a is greater than b;

[0031] Sum the values ​​that have not been adjusted by dynamic weights and the values ​​that have been adjusted by dynamic weights to obtain the unwrapped value and the wrapped value;

[0032] Determine whether the difference between the unwrapped value and the wrapped value is less than the difference threshold; if yes, keep the order of the candidate policy texts; if no, sort the policy text at the first position in the matching index to the last position.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. In the present invention, by combining the two-stage retrieval mechanism of reverse index and embedded retrieval, candidate policy texts are firstly quickly screened out through the reverse index, and then several policy texts and demand questions are converted into two high-dimensional vectors and compared, and fine sorting is performed according to the comparison results. Several candidate policy texts finely sorted by the large language model are used as reference for answering, which can improve the relevance and accuracy of the retrieval results.

[0035] 2. In the present invention, dynamic weight adjustment is added to the output layer of the llama3 model, and contextual connection and optimized matching index are performed according to multiple demand questions raised by users, so that subsequent demand questions can be more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a connection diagram of the modules in the present invention;

[0038] Figure 2 A schematic diagram of the working principle of the policy question answering system based on a large language model, inverted index and embedded retrieval of the present invention; DETAILED DESCRIPTION

[0039] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] See also Figure 1-Figure 2, the first aspect of the present invention provides a policy question-answering system based on a large language model, a reverse index and an embedded retrieval, including: a data preprocessing module, a reverse index module, an embedded retrieval module, a retrieval management module, a question-answering generation module and a user interface module;

[0041] See also Figure 1 As shown, the data preprocessing module is connected in communication with the reverse index module, the reverse index module is connected in communication with the embedded retrieval module, the reverse index module and the embedded retrieval module are both connected in communication with the retrieval management module, the retrieval management module is connected in communication with the question and answer generation module, and the question and answer generation module is connected in communication with the user interface module;

[0042] Data preprocessing module: used to obtain policy texts and demand questions; and to clean the text in the policy text; the cleaning methods include word segmentation, stop word removal and stem extraction;

[0043] Reverse index module: Create a reverse index based on the cleaned policy text, and build a term-document inverted list based on the text index result; and match the reverse index of the demand question with the term-document inverted list to obtain several candidate policy texts;

[0044] Embedding retrieval module: The word embedding module transforms the demand question into a high-dimensional vector of the question; the word embedding model transforms the cleaned policy text into a high-dimensional policy vector, and constructs a policy vector index;

[0045] Retrieval management module: matches the problem high-dimensional vector with several policy high-dimensional vectors to obtain a matching index; wherein the matching index is composed of several candidate policy texts with a sequence;

[0046] Question and answer generation module: The large language model generates answers corresponding to the required questions based on the matching index; the large language model is built based on the artificial intelligence model;

[0047] User interface module: used to input requirement questions; and display answers to the requirement questions and corresponding policy texts.

[0048] The term-document inverted list mentioned above is a way to map terms (words or phrases) in a document to a list of documents containing these terms, so as to quickly find all documents containing a certain term. The terms are sorted according to the number of occurrences. The documents and positions where each term appears are recorded in the inverted list to achieve fast retrieval.

[0049] Specifically, the large language model is built based on the artificial intelligence model, including:

[0050] Obtain quality fine-tuning datasets and training datasets, build a large language model to be trained through the llama3-chinese-chat-7B model, add dynamic weight adjustment to the output layer of the llama3-chinese-chat-7B model, and introduce the self-attention mechanism;

[0051] The weight matrix in the large language model to be trained is decomposed into low rank. The weight matrix W can be decomposed into two smaller matrices A and B by low rank decomposition, and W is approximately equal to A×B. The ranks of matrices A and B are much smaller than the dimensions of the original weight matrix. This decomposition greatly reduces the number of parameters that need to be trained.

[0052] Insert a low-rank adapter into the weight matrix of the model’s self-attention layer;

[0053] The large language model to be trained is trained with the training data set, and the trained large language model is tested with the quality fine-tuning data set. The trained large language model is adjusted according to the test results, and finally a large language model is obtained whose input is the required question and whose output is the answer without dynamic weight adjustment and the answer after dynamic weight adjustment.

[0054] Furthermore, a term-document inverted list is constructed based on the text index results, including:

[0055] Obtain the text index result of the reverse index of the cleaned policy text, count the number of times several terms appear in the text index result, and obtain the document ID of the policy text where the corresponding term is located and the position information in the corresponding policy text; draw several terms, the document ID of the policy text where the term corresponds to, and the position in the corresponding policy text into an arrangement table to obtain a term-document inverted list.

[0056] For example, the following policy text exists:

[0057] Doc1: "The maternity leave for women in Beijing is 128 days";

[0058] Doc2: "Maternity leave for women in Shanghai is 128 days";

[0059] Doc3: "The maternity leave for women in Guangdong Province is 158 days";

[0060] Doc4: "The maternity leave for women in Inner Mongolia Autonomous Region is 158 days";

[0061] Doc5: "The maternity leave for women in Guangxi Zhuang Autonomous Region is 190 days";

[0062] Doc6: "Maternity leave for women in Ningxia Hui Autonomous Region is 180 days";

[0063] Doc7: "The nursing leave for women in Ningxia Hui Autonomous Region is 15 days";

[0064] Doc8: "The nursing leave for men in Guangxi Zhuang Autonomous Region is 18 days";

[0065] Then the constructed term-document inverted list is:

[0066] Beijing: [Doc1]

[0067] Shanghai: [Doc2]

[0068] Guangdong Province: [Doc3]

[0069] Inner Mongolia Autonomous Region: [Doc4]

[0070] Guangxi Zhuang Autonomous Region: [Doc5, Doc8]

[0071] Ningxia Hui Autonomous Region: [Doc6, Doc7]

[0072] Female: [Doc1, Doc2, Doc3, Doc4, Doc5, Doc6, Doc7]

[0073] Maternity leave: [Doc1, Doc2, Doc3, Doc4, Doc5, Doc6]

[0074] Nursing leave: [Doc7, Doc8]

[0075] Male: [Doc8]

[0076] 128 days: [Doc1, Doc2]

[0077] 158 days: [Doc3, Doc4]

[0078] 190 days: [Doc5]

[0079] 180 days: [Doc6]

[0080] 15 days: [Doc7]

[0081] 18 days: [Doc8];

[0082] It should be noted that this example simply cites the policy text for the sake of simplicity, and therefore omits the corresponding position in the policy text. The following illustrative content will omit the same content and will not be repeated here.

[0083] After the demand question is inverted indexed, it is matched with the term-document inverted list, including:

[0084] Clean the demand questions, and reverse index the cleaned demand questions to obtain the question index results;

[0085] Matching the terms in the question index result with the terms in the term-document inverted list to obtain matching terms, and marking several policy text document IDs recorded by the matching terms in the term-document inverted list as screening policy texts;

[0086] The policy texts with the highest relevance are selected from the screened policy texts and marked as candidate policy texts.

[0087] For example, the demand question is: How many days of maternity leave are there for women in Ningxia Hui Autonomous Region? The content of the cleaned index for this question is: [Ningxia Hui Autonomous Region, women, maternity leave]; and the matching term is [Ningxia Hui Autonomous Region, women, maternity leave], and the policy text corresponding to the matching term is:

[0088] Ningxia Hui Autonomous Region: [Doc6, Doc7];

[0089] Female: [Doc1, Doc2, Doc3, Doc4, Doc5, Doc6, Doc7];

[0090] Maternity leave: [Doc1, Doc2, Doc3, Doc4, Doc5, Doc6];

[0091] It can be seen that the filtered text is Doc1, Doc2, Doc3, Doc4, Doc5, Doc6, Doc7;

[0092] Doc6 has the highest relevance, so Doc6 is the candidate policy text.

[0093] Furthermore, the problem high-dimensional vector is matched with several policy high-dimensional vectors, including:

[0094] Calculate the Euclidean distance between the problem high-dimensional vector and several policy high-dimensional vectors respectively, and mark them as Li; where i = 1, 2, ... n, n is the number of policy high-dimensional vectors;

[0095] The policy texts corresponding to the Euclidean distances Li are sorted from large to small to obtain a matching index.

[0096] It should be noted that since there is only one candidate policy text, Doc6, in the above exemplary content, this step may not be performed in this example. If there are multiple candidate policy texts, the problem high-dimensional vector and several policy high-dimensional vectors may be matched.

[0097] In another real-time example, a policy question-answering system based on a large language model, inverted indexing and embedded retrieval also includes an implementation optimization module; the real-time optimization module is communicated with the question-answering management module.

[0098] The user interface module in this embodiment: clicks to view the candidate policy text in the matching index and provides feedback on the answer corresponding to the requirement question; wherein the feedback result includes satisfaction and dissatisfaction;

[0099] Specifically, an optimization module is implemented: for obtaining feedback results of users' answers corresponding to a number of demand questions; and for adjusting the order of a number of candidate policy texts in the matching index based on a number of feedback results.

[0100] By adjusting the order of candidate policy texts, the accuracy of answers to subsequent demand questions can be further improved.

[0101] Furthermore, the order of several candidate policy texts in the matching index is adjusted based on several feedback results, including:

[0102] Obtain the feedback results corresponding to the answers to several demand questions that have not been dynamically weighted and those that have been dynamically weighted; assign satisfaction and dissatisfaction in the feedback results to 1 and 0 respectively

[0103] Sum the values ​​that have not been adjusted by dynamic weights and the values ​​that have been adjusted by dynamic weights to obtain the unwrapped value and the wrapped value;

[0104] Determine whether the difference between the unwrapped value and the wrapped value is less than the difference threshold 1; if yes, keep the order of the candidate policy texts; if no, sort the policy text at the first position in the matching index to the last position.

[0105] The working principle of the present invention is as follows: the user inputs the demand question through the user interface module, and the reverse index module performs reverse indexing on the demand question to obtain a number of candidate policy texts;

[0106] The candidate policy texts are converted into several high-dimensional policy vectors through the embedding retrieval module, and the demand questions are converted into high-dimensional question vectors through the word embedding module;

[0107] The similarities between the problem high-dimensional vector and several policy high-dimensional vectors are calculated respectively to obtain several similarity values, and several candidate policy texts corresponding to the several similarity values ​​are sorted from large to small to obtain a sorted list; the large language model generates the answer corresponding to the required question based on the policy text ranked highest in the sorted list.

[0108] It should be noted that the policy text is obtained; the text in the policy text is cleaned; a reverse index is created based on the cleaned policy text, and a term-document inverted list is constructed based on the text index result; the cleaned policy text is converted into a policy high-dimensional vector through a word embedding model, and a policy vector index is constructed; these steps have been completed before the user enters the requirement question.

[0109] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. Policy question answering system based on large language model, inverted index and embedding retrieval, It is characterized in that include: Data preprocessing module, reverse indexing module, embedded retrieval module, retrieval management module, question and answer generation module and user interface module; The data preprocessing module is used to obtain policy texts and demand issues; and to clean the text in the policy text; wherein the cleaning methods include word segmentation, stop word removal and stem extraction; The reverse index module: creates a reverse index based on the cleaned policy text, and builds a term-document inverted list based on the text index result; and, after reverse indexing the demand question, matches it with the term-document inverted list to obtain a number of candidate policy texts; The embedding retrieval module: converts the demand question into a high-dimensional question vector through a word embedding module; converts the cleaned policy text into a high-dimensional policy vector through a word embedding model; The retrieval management module: matches the problem high-dimensional vector with a plurality of policy high-dimensional vectors to obtain a matching index; wherein the matching index is composed of a plurality of candidate policy texts with a sequence; The question-answer generation module: the large language model generates answers corresponding to the required questions according to the matching index; wherein the large language model is established based on the artificial intelligence model; The user interface module is used for inputting requirement questions and displaying answers to the requirement questions and corresponding policy texts.

2. The policy question answering system based on large language model, inverted index and embedded retrieval according to claim 1, It is characterized in that The large language model is established based on an artificial intelligence model, including: Obtain quality fine-tuning datasets and training datasets, build a large language model to be trained through the llama3 model, add dynamic weight adjustment to the output layer of the llama3 model, and introduce the self-attention mechanism; Perform low-rank decomposition on the weight matrix of the large language model to be trained; Insert a low-rank adapter at a specific layer of the model; the specific layer is any dense layer of the model, including the weight matrix of the self-attention layer; The large language model to be trained is trained through the training data set, and the trained large language model is tested through the quality fine-tuning data set. The trained large language model is adjusted according to the test results, and finally the trained large language model is obtained.

3. The policy question answering system based on large language model, inverted index and embedded retrieval according to claim 1, It is characterized in that Build a term-document inverted list based on the text index results, including: Obtain the text index result of the reverse index of the cleaned policy text, count the number of times several terms appear in the text index result, and obtain the document ID of the policy text where the corresponding term is located and the position information in the corresponding policy text; draw several terms, the document ID of the policy text where the term corresponds to, and the position in the corresponding policy text into an arrangement table to obtain a term-document inverted list.

4. The policy question answering system based on large language model, inverted index and embedded retrieval according to claim 1, It is characterized in that The reverse indexing of the requirement question is matched with the term-document inverted list, including: Clean the demand questions, and reverse index the cleaned demand questions to obtain the question index results; Matching the terms in the question index result with the terms in the term-document inverted list to obtain matching terms, and marking several policy text document IDs recorded by the matching terms in the term-document inverted list as screening policy texts; The policy texts with the highest relevance are selected from the screened policy texts and marked as candidate policy texts.

5. The policy question answering system based on large language model, inverted index and embedded retrieval according to claim 1, It is characterized in that The matching of the problem high-dimensional vector and the plurality of policy high-dimensional vectors includes: Calculate the Euclidean distance between the problem high-dimensional vector and several policy high-dimensional vectors respectively, and mark them as Li; where i = 1, 2, ... n, n is the number of policy high-dimensional vectors; The policy texts corresponding to the Euclidean distances Li are sorted from large to small to obtain a matching index.

6. The policy question answering system based on large language model, inverted index and embedded retrieval according to claim 1, It is characterized in that The system also includes an implementation optimization module, wherein the user interface module: clicks to view the candidate policy text in the matching index and provides feedback on the answer corresponding to the required question; The feedback results include satisfaction and dissatisfaction; The implementation optimization module is used to obtain feedback results of users' answers corresponding to a number of demand questions; and to adjust the order of a number of candidate policy texts in the matching index based on the feedback results.

7. The policy question answering system based on large language model, inverted index and embedded retrieval according to claim 1, It is characterized in that The order of several candidate policy texts in the matching index is adjusted based on several feedback results, including: Obtain the feedback results corresponding to the answers to several demand questions that have not been dynamically weighted and those that have been dynamically weighted; assign the satisfaction and dissatisfaction in the feedback results to a and b respectively, and a is greater than b; Sum the values ​​that have not been adjusted by dynamic weights and the values ​​that have been adjusted by dynamic weights to obtain the unwrapped value and the wrapped value; Determine whether the difference between the unwrapped value and the wrapped value is less than the difference threshold; if yes, keep the order of the candidate policy texts; if no, sort the policy text at the first position in the matching index to the last position.

Images