A government affair intelligent question and answer method

Abstract

The application provides a kind of intelligent government affair question and answer method, comprising the following steps: S1, government affair file is used to build government affair knowledge base and question and answer data set;S2, through the instruction following function in open source model, ask the knowledge base file, generate government affair question and answer training data;S3, through the pre-training language model of well-trained, based on the policy problem of user query, from the government affair knowledge database, the relevant policy file is matched in coarse granularity;S4, the policy file is cut into policy paragraph, and the question and policy paragraph are input into the pre-training language model of well-trained, guide model and the most relevant policy paragraph of question match;S5, the question and policy paragraph are constructed into prompt, and the prompt is input into the large language model of well-trained, guide the large language model of well-trained to generate the answer of question.The application can achieve diversity generation on the basis of accurate answer, and users can ask the policy content they care about, and obtain the answer of the latest policy content in real time.

Description

Technical Field

[0001] This invention relates to a government affairs intelligent question-answering method, belonging to the fields of natural language processing and deep learning technology. Background Technology

[0002] Domain-specific question answering is a practical and challenging natural language processing task.

[0003] In existing technologies, researchers have built open-source Dense Passage Retrieval (DPR) based on pre-trained BERT models. Through contrastive learning and carefully designed negative sampling strategies, DPR, after training, has become a more robust baseline for domain question answering. Others have employed innovative negative sampling methods to train dual-encoder retrieval models (DPRs), resulting in RocketQA. RocketQA introduces multi-GPU training across batches of negative examples, or utilizes a trained cross-encoder to select high-confidence negative examples from the top-k retrieval of the dual encoder as denoising hard negative examples. Simultaneously, RocketQA constructs new training examples from a set of unlabeled questions and synthesizes paragraph labels using the cross-encoder. Still others have extended RocketQA by combining dynamic list distillation with a joint training method for the retrieval and reordering mechanisms. However, these existing technologies only employ the retrieval model and do not utilize a reader to generate user-understandable answers.

[0004] Large Language Models (LLMs) are gaining increasing importance in the field of artificial intelligence. Powerful models such as ChatGPT and LLaMA have had a significant impact on many areas, including society, business, and research. However, LLMs still face serious factual environment problems in domain applications; that is, due to a lack of knowledge about the external world, LLMs often generate false statements with inaccurate factual information. While some work has been done on domain applications, some literature mentions that fine-tuning LLM instructions can successfully solve some domain-specific question-answering problems. However, because LLMs do not update over time, in real-time domains such as government affairs, the accuracy of directly generated answers may be unsatisfactory. There are also issues such as requiring excessive computational resources, generating sentences with limited structure, and insufficient content. Summary of the Invention

[0005] To address the problems of existing technologies and achieve accurate, controllable, and diverse intelligent question answering for government affairs, this invention fully learns from a pre-trained language model. First, a government affairs knowledge database is constructed. Then, based on this database, the instruction-following capability of a large language model is used to generate a series of training datasets. Subsequently, a policy document retrieval component, a paragraph matching component, and a reading comprehension component are trained based on these training datasets. The technical solution of this invention is described below.

[0006] A government affairs intelligent question-answering method includes the following steps:

[0007] S1. Perform data cleaning, including deduplication and retaining valid documents, in order to establish a government knowledge database;

[0008] S2. Using the instruction follow-up function in the currently available open-source models, ask questions to the knowledge base files to generate a government affairs question and answer training dataset.

[0009] S3. Using a pre-trained language model, relevant policy documents are coarsely matched from the government knowledge database based on the policy questions queried by the user.

[0010] S4. Segment the policy document into policy paragraph data, and input the problem and policy paragraph data into the pre-trained language model to guide the model to match the policy paragraph most relevant to the problem;

[0011] S5. Construct prompts from the question and policy paragraph data, and input the prompts into the trained large language model to guide the trained large language model to generate answers to the questions.

[0012] The detailed steps are described below.

[0013] 1. For the main step S1, establishing a government knowledge database includes the following sub-steps:

[0014] S11: Remove invalid information from the data; remove private data from the data, such as ID card numbers, mobile phone numbers, email addresses, etc.; retain only policy documents with a content length greater than 100 characters; use open-source hash deduplication methods to deduplicate policy documents;

[0015] S12. Collect and retain the provincial, municipal, and county-level hierarchical structure, source institutions, and time tags from valid policy documents;

[0016] 2. For the main step S2, the question-answering training data includes the following sub-steps:

[0017] S21. Randomly select policy documents from the policy knowledge base and divide the selected policy documents into paragraphs according to the document structure;

[0018] S22. Set prompts and use the instruction following capability of the large language model to ask questions about the paragraph described in S21, generating question data in the question-and-answer data.

[0019] S23. Set prompts and use the instruction-following capability of the large language model to answer the questions in S22 based on the paragraphs in S21, generating answer data in the question-and-answer data.

[0020] S24. Integrate the paragraphs in S21, the question data in S22, and the answer data in S23, and save them as a training dataset for model training.

[0021] 3. The training of the pre-trained language model described in main step S3 includes the following sub-steps:

[0022] S31. Pre-train an open-source BERT model on the Chinese ad-hoc retrieval dataset T2ranking;

[0023] S32. From the government affairs question-and-answer dataset generated in step S2, select training datasets, including: dividing the documents in the question-and-answer dataset according to prefecture-level cities, obtaining the document titles, and clustering the document titles using the Ward clustering method to obtain similar documents; for similar documents, selecting questions with diversity; using the BERT model trained in S31, vectorizing the questions, obtaining the similarity matrix between questions based on cosine similarity, and then using hierarchical clustering again to obtain questions with high similarity. The similarity acquisition method is as follows:

[0024]

[0025] in, and For the question text, This refers to the BERT model that encodes the user's query in DPR retrieval. [CLS] is the start character for the text encoding in the BERT model, and the denominator contains... It is the vector norm.

[0026] Based on the clustering problem, samples are taken from each category of the problem, and the corresponding documents are obtained as the actual training dataset.

[0027] S33. Based on the training dataset obtained in S32, the open-source Dense Paragraph Retrieval (DPR) model is trained and fine-tuned to obtain the trained model.

[0028] S34. This system focuses on meeting the geographical, time-sensitive, and institutional needs of government Q&A system users. To this end, the inventors constructed a hierarchical tree index based on province-city-district-issuing agency-issuance time, as detailed below:

[0029] Based on the actual place names' province-city-county-street structure, a hierarchical tree-like node structure is constructed. Nodes have multiple attributes, such as time attributes, place name attributes, and organization attributes, all of which are lists. Among them, place name attributes and organization attributes have inheritance characteristics, that is, child nodes will inherit the corresponding attribute values ​​of parent nodes. For example, the Putian City node is a child node of the Fujian Province node. The Fujian Province node has the place name attribute "Fujian". Therefore, the Putian City node has the place name attributes "Putian" and "Fujian" in addition to "Putian".

[0030] Secondly, based on the organization tag information in the document dataset, new child nodes are added under the corresponding organization and place name nodes, which inherit the place name information and organization attributes of the parent node.

[0031] In addition, time nodes are constructed. Time nodes are leaf nodes of the tree index, inheriting the place name and organization attributes of the parent node, and also having time attributes.

[0032] S35. Use the pre-trained language model trained in S33 to pre-encode government documents using vectors, as shown in the following formula:

[0033]

[0034] Among them, doc refers to government documents. This refers to the BERT model used in DPR retrieval to encode government documents. [CLS] represents the start character for text encoding in the BERT model. This is a vector representation of government documents.

[0035] S36. Place the corresponding vector in the leaf node of the corresponding hierarchical tree index according to the document's tags;

[0036] S37. Using the pre-trained language model trained in S33, the user query is vector-encoded using the following formula:

[0037]

[0038] Where q represents the government service question queried by the user. This refers to the BERT model used in DPR retrieval to encode the user's query question. [CLS] represents the start character for the text encoding in the BERT model. This is a vector representation of government documents.

[0039] Based on the dual-tower model, the similarity between the question and the government document is obtained, as shown in the following formula:

[0040]

[0041] Among them, E q Ed These are represented as questions and documents, respectively. This represents the similarity score between the question and the document.

[0042] The model is trained using an in-batch negative approach, selecting more difficult samples as negative samples. For each question, BM25 negative samples are selected, i.e., irrelevant documents ranked high by BM25. For a given question, relevant documents are selected as positive samples, and the BM25 data for that question, along with relevant documents from other questions, are selected as negative samples. ListWise loss is used for training, as shown in the following formula.

[0043]

[0044] in For the i-th problem in the training process, For the i-th user, query the relevant paragraphs corresponding to the question. For the i-th user, the irrelevant questions corresponding to the query question. Let j represent the j-th unrelated question. The similarity sim is calculated using the vector dot product method. The notation represents the loss function generated based on the above information.

[0045] S38. Use word-based matching methods to obtain documents with high similarity and precise matching.

[0046] 4. Step S4 training method includes:

[0047] S41. From the government affairs Q&A dataset generated in step S2, filter out policy paragraphs and question training datasets;

[0048] S42. Using the pre-trained language model Roberta-large-wwm classification model, the questions generated for the corresponding paragraphs are taken as positive samples, while other paragraphs in the document containing the questions are taken as negative samples. The question-evidence pairs are concatenated and then trained using a Cross Encoder structure. The training loss is shown below:

[0049]

[0050] in The score predicted by the model. Relevance tags for paragraphs and questions;

[0051] S43. Divide the policy articles provided in the S3 phase into paragraphs;

[0052] S44. Concatenate the paragraph and the question and input them into the pre-trained language classification model obtained in S42 to obtain the similarity between the paragraph and the question;

[0053] S45. Return to the most similar paragraph in S44;

[0054] 5. Step S5 specifically includes the following:

[0055] S51. From the government affairs question and answer dataset generated in step S2, filter out policy paragraphs, questions, and answer training datasets;

[0056] S52. The obtained paragraphs, questions, and answers are concatenated into a prompt, which is then input into the pre-trained large language model MindLLM for fine-tuning and training of the government affairs large language model. MindLLM is the Mingde large language model released by Beijing Institute of Technology.

[0057] S53. Use the paragraph data obtained from step S4 as evidence and combine it with the question to form a prompt.

[0058] S54. Input the prompt into the pre-trained large language model obtained in S52, use the reasoning and generation capabilities of the large language model to generate an answer, and return it to the user.

[0059] Beneficial effects

[0060] This invention constructs a government affairs question-and-answer database and generates a batch of training datasets based on the instruction-following capabilities of a large language model. It then trains pre-trained language models for policy document recall, paragraph matching, and reading comprehension, without adding extra parameters or computational costs. This achieves efficient, accurate, and real-time updated answers. Users can submit various questions about government affairs through the system, which accurately understands the questions and provides detailed answers. Based on deep learning and the training of a large language model, it enables efficient government affairs consultation services. Compared to existing technologies, this invention can obtain government affairs question-and-answer information more in real-time, accurately, and quickly, thus bringing progress to the field of question answering. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the intelligent government affairs question-and-answer process of the present invention;

[0062] Figure 2 This is a flowchart illustrating the government affairs intelligent knowledge base and question-and-answer dataset of the present invention.

[0063] Figure 3 This is a schematic diagram illustrating the policy document recall for this invention;

[0064] Figure 4 This is a schematic diagram of paragraph matching according to the present invention;

[0065] Figure 5This is a schematic diagram illustrating the reading comprehension aspect of the present invention. Detailed Implementation

[0066] To make the present invention clearer, the technical solution of the present invention will be described in detail and completely below with reference to the accompanying drawings and specific embodiments. It should be noted that the specific embodiments described in the present invention are only some embodiments of the present invention, not all embodiments, and do not constitute any limitation on the present invention.

[0067] See the instruction manual appendix Figure 1 This invention discloses a government affairs intelligent question-and-answer method, which includes the following steps:

[0068] S1. Perform data cleaning, including deduplication and retaining valid documents, in order to establish a government knowledge database;

[0069] S2. Using the instruction follow-up function in the currently available open-source models, ask questions to the knowledge base files to generate government affairs question and answer training data.

[0070] S3. Using a pre-trained language model, relevant policy documents are coarsely matched from the government knowledge database based on the policy questions queried by the user.

[0071] S4. Divide the policy document into policy paragraphs, and input the problem and policy paragraphs into the pre-trained language model to guide the model to match the policy paragraph most relevant to the problem;

[0072] S5. Construct prompts from the questions and policy paragraphs, and input the prompts into the trained large language model to guide the trained large language model to generate answers to the questions.

[0073] The above five steps include the following sub-steps.

[0074] See the instruction manual appendix Figure 1 The construction of the government knowledge database in step S1 includes:

[0075] S11. Clean all policy document data, including data deduplication, retaining only valid policy documents, including:

[0076] (1) Data cleaning includes removing invalid information from the data;

[0077] (2) Remove private data from the data, including ID card number, mobile phone number, email address and other private information;

[0078] (3) Only retain policy documents with a content length greater than 100 characters;

[0079] (4) Use open-source hash deduplication methods to deduplicat policy documents;

[0080] S12. Compile and retain the provincial, municipal, and county-level structure, source institutions, and time tags of policy documents from valid policy documents;

[0081] 2. The specific steps for generating question-answering training data in S2 are detailed in the appendix of the instruction manual. Figure 2 :

[0082] S21. Randomly select policy documents from the policy knowledge base and divide the selected policy documents into paragraphs according to the document structure.

[0083] S22. Set prompts and use the instruction following capability of the large language model to ask questions about the paragraph described in S21, generating question data in the question-and-answer data.

[0084] S23. Set prompts and use the instruction-following capability of the large language model to answer the questions in S22 based on the paragraphs in S21, generating answer data in the question-and-answer data.

[0085] S24. Integrate the paragraphs in S21, the question data in S22, and the answer data in S23, and save them as a training dataset for model training.

[0086] 3. In step S3, the training of the pre-trained language model includes, as detailed in the appendix to the specification. Figure 3 , 4 5:

[0087] S31. Pre-train an open-source BERT model on the Chinese ad-hoc retrieval dataset T2ranking;

[0088] S32. From the government affairs Q&A data generated in step S2, filter the training dataset, including:

[0089] (1) First, the files in the dataset are divided according to the province, and then further divided according to the prefecture-level city. The titles of the documents are obtained, and the document titles are clustered by the Ward clustering method to obtain similar documents.

[0090] (2) For similar documents, select those with diversity;

[0091] (3) Using the BERT model trained in S32, the questions are vectorized, and the similarity matrix between questions is obtained based on cosine similarity. Then, hierarchical clustering is used again to obtain questions with high similarity. The similarity is obtained as follows:

[0092]

[0093] in, and For the question text, This refers to the BERT model that encodes the user's query in DPR retrieval. [CLS] is the start character for the text encoding in the BERT model, and the denominator contains... It is the vector norm.

[0094] (4) Finally, based on the clustering problem, the problems of each category are sampled and the corresponding documents are obtained as the actual training dataset.

[0095] S33. Based on the training dataset obtained in S32, the open-source Dense Paragraph Retrieval (DPR) model is trained and fine-tuned to obtain the trained model.

[0096] S34. To select diversity and meet the needs of government Q&A system users in terms of location, timeliness, and organization, specific judgment rules are executed, including queries based on place name keywords. To this end, a hierarchical tree index is constructed based on province-city-district-issuing agency-issuance time; the operation is as follows:

[0097] Based on the actual place names' province-city-county-street structure, a hierarchical tree-like node structure is constructed. Nodes have multiple attributes, including time attributes, place name attributes, and organization attributes, all of which are lists. Among them, place name attributes and organization attributes have inheritance characteristics, meaning that child nodes will inherit the corresponding attribute values ​​of their parent nodes. For example, the Putian City node is a child node of the Fujian Province node. The Fujian Province node has the place name attribute "Fujian," therefore the Putian City node's place name attributes include both "Putian" and "Fujian."

[0098] Secondly, based on the organization label information in the document dataset, a new child node is added under the leaf node of the corresponding organization. This child node inherits the place name information and organization attributes of the parent node.

[0099] In addition, time nodes are constructed. Time nodes are leaf nodes of the tree index, inheriting the location and organization attributes of the parent node, and also having a time attribute.

[0100] S35. Use the pre-trained language model trained in S33 to pre-encode government documents using vectors, as shown in the following formula:

[0101]

[0102] Among them, doc refers to government documents. This refers to the BERT model used in DPR retrieval to encode government documents. [CLS] represents the start character for text encoding in the BERT model. This is a vector representation of government documents.

[0103] S36. Place the corresponding vector in the leaf node of the corresponding hierarchical tree index according to the document's tags;

[0104] S37. Using the pre-trained language model trained in S33, the user query is vector-encoded using the following formula:

[0105]

[0106] Where q represents the government service question queried by the user. This refers to the BERT model used in DPR retrieval to encode the user's query question. [CLS] represents the start character for the text encoding in the BERT model. This is a vector representation of government documents.

[0107] S36. Place the corresponding vector in the leaf node of the corresponding hierarchical tree index according to the document's tags;

[0108] S37. Using the pre-trained language model trained in S33, the question is vector-encoded, and based on the question representation, document vectors similar to the question are matched.

[0109] Based on the dual-tower model, the similarity between the question and the government document is obtained, as shown in the following formula:

[0110]

[0111] Where Eq and Ed represent the question and document, respectively. This represents the similarity score between the question and the document.

[0112] The model is trained using an in-batch negative approach, selecting more difficult samples as negative samples. For each question, BM25 negative samples are selected, i.e., irrelevant documents ranked highly by BM25. For a given question, relevant documents are selected as positive samples, and BM25 negative samples for that question, along with relevant documents from other questions, are selected as negative samples. ListWise loss is used for training, as shown in the following formula.

[0113]

[0114] in For the i-th problem in the training process, For the i-th user, query the relevant paragraphs corresponding to the question. For the i-th user, the irrelevant questions corresponding to the query question. Let j represent the j-th unrelated question. The similarity sim is calculated using the vector dot product method. The notation represents the loss function generated based on the above information.

[0115] S38. Using an inverted index, a word-based matching method is used to obtain documents with high similarity and precise matching.

[0116] 4. In step S4, the problem and policy paragraph are input into the pre-trained language model, and the training method that guides the model to match the policy paragraph most relevant to the problem includes:

[0117] S41. From the government affairs Q&A data generated in step S2, filter out policy paragraphs and question training datasets;

[0118] S42. Using the pre-trained Roberta-large-wwm classification model, the question generated for the corresponding paragraph is considered a positive sample, while other paragraphs in the document containing the question are considered negative samples. The question-evidence pairs are concatenated and trained using a Cross Encoder architecture, where the training loss L... s As shown below:

[0119]

[0120] in The score predicted by the model. These are relevance tags for paragraphs and questions.

[0121] S43. In the reasoning stage, the policy article provided in stage S3 is divided into paragraphs;

[0122] S44. Concatenate the paragraph and the question as input, and feed them into the pre-trained language classification model obtained in S42 to obtain the similarity between the paragraph and the question;

[0123] S45. Return to the most similar paragraph;

[0124] 5. In step S5, the method for guiding the trained large language model to generate answers to questions includes:

[0125] S51. From the government affairs Q&A data generated in S2, select policy paragraphs, questions, and answers to train the dataset;

[0126] S52. The obtained paragraphs, questions, and answers are concatenated into a prompt, which is then input into the pre-trained large language model MindLLM for fine-tuning and training of the government affairs large language model.

[0127] S53. Use the paragraph data obtained from S4 as evidence and combine it with the question to form a prompt;

[0128] S54. Input the prompt into the pre-trained large language model obtained in S52, use the reasoning and generation capabilities of the large language model to generate an answer, and return it to the user.

[0129] The present invention has been described in detail above through embodiments. It should be understood that, without departing from the spirit and essence of the present invention and without making any inventive effort, those skilled in the art can modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions or further extensions to some or all of the technical features therein, and all other embodiments or technical solutions obtained through these modifications, substitutions or further extensions fall within the protection scope of the present invention.

Claims

1. A government affairs intelligent question-answering method, characterized in that, The intelligent government Q&A method includes the following steps: S1. Conduct data cleaning to establish a government knowledge database; S2. Using the instruction follow-up function in the currently available open-source models, ask questions to the knowledge base files to generate a government affairs question and answer training dataset. S3. Using a pre-trained language model, relevant policy documents are coarsely matched from the government knowledge database based on the policy questions queried by the user. S4. Segment the policy document into policy paragraph data, and input the problem and policy paragraph data into the pre-trained language model to guide the model to match the policy paragraph most relevant to the problem; S5. Construct prompts from the questions and policy paragraphs, and input the prompts into the trained large language model to guide the trained large language model to generate answers to the questions; S3 includes: S31. Pre-train an open-source BERT model on the Chinese ad-hoc retrieval dataset T2ranking; S32. Select a training dataset from the government affairs Q&A data generated in S2; S33. Based on the training dataset obtained in S32, the open-source Dense Paragraph Retrieval (DPR) model is trained and fine-tuned to obtain the trained model. S34. Based on the province-city-county-street structure of actual place names, construct a hierarchical tree node. Nodes have time attributes, place name attributes, and organization attributes, all of which are lists. Among them, the place name attributes and organization attributes have inheritance characteristics. Based on the organization tag information in the document dataset, add new child nodes under the corresponding organization and place name nodes. These child nodes inherit the place name information and organization attributes of the parent node. Construct time nodes. Time nodes are the leaf nodes of the tree index. They inherit the place name and organization attributes of the parent node and also have a time attribute. S35. Use the pre-trained language model trained in S33 to pre-encode the document into vectors, as shown in the following formula: This involves obtaining feature vectors from documents used in training; where doc represents government documents. This refers to the BERT model used in DPR retrieval to encode government documents. [CLS] represents the start character for text encoding in the BERT model. A vector representation of government documents; S36. Place the corresponding vector in the leaf node of the corresponding hierarchical tree index according to the document's tags; S37. Using the pre-trained language model trained in S33, the user query is vector-encoded using the following formula: Where q represents the government service question queried by the user. This refers to the BERT model used in DPR retrieval to encode the user's query question. [CLS] represents the start character for the text encoding in the BERT model. A vector representation of government documents; Based on the dual-tower model, the similarity between the question and the government document is obtained, as shown in the following formula: Among them, E q E d These are represented as questions and documents, respectively. This represents the similarity score between the question and the document. The model is trained using an in-batch negative approach. For each question, relevant documents are selected as positive samples, the BM25 of that question is selected as negative samples, and relevant documents from other questions are selected as negative samples. ListWise loss is used for training, as shown in the following formula: in, For the i-th problem in the training process, For the i-th user, query the relevant paragraphs corresponding to the question. For the i-th user, the irrelevant questions corresponding to the query question. To represent the j-th unrelated question, the similarity sim is calculated using the vector dot product method. The notation representing the loss function generated based on the above information; S38. Use word-based matching methods to obtain documents with high similarity and precise matching.

2. The government affairs intelligent question-answering method according to claim 1, characterized in that, S1 includes the following sub-steps: S11. Clean all policy document data, including deduplication, and retain valid policy documents; S12. Compile and retain the provincial, municipal, and county-level structure, source institutions, and time tags of policy documents from valid policy documents.

3. The intelligent question-and-answer method for government affairs according to claim 2, characterized in that, S2 includes the following sub-steps: S21. Randomly select policy documents from the policy knowledge base and divide the selected policy documents into paragraphs according to the document structure; S22. Set prompts and use the instruction following capability of the large language model to ask questions about the paragraph described in S21, generating question data in the question-and-answer data. S23. Set prompts and use the instruction-following capability of the large language model to answer the questions in S22 based on the paragraphs in S21, generating answer data in the question-and-answer data. S24. Integrate the paragraphs in S21, the question data in S22, and the answer data in S23, and save them as a training dataset for model training.

4. The intelligent question-and-answer method for government affairs according to claim 3, characterized in that, S4 includes: S41. From the government affairs question and answer dataset generated in S2, filter out policy paragraphs and question training datasets; S42. Using the pre-trained language model Roberta-large-wwm, the questions generated for the corresponding paragraphs are taken as positive samples, while other paragraphs in the document containing the questions are taken as negative samples. The question-evidence pairs are concatenated and then trained using a CrossEncoder structure. The training loss is shown below: in The score predicted by the model. Relevance tags for paragraphs and questions; S43. Divide the policy article provided in stage S3 into paragraphs; S44. The paragraph and the question are concatenated and input into the pre-trained language classification model obtained in S42 to obtain the similarity between the paragraph and the question; S45. Return to the most similar paragraph in S44.

5. The government affairs intelligent question-answering method according to claim 4, characterized in that, S5 includes: S51. From the government affairs question and answer dataset generated in S2, filter out policy paragraphs, questions, and answer training datasets; S52. The obtained paragraphs, questions and answers are concatenated into prompts, which are then input into the pre-trained MindLLM large language model for fine-tuning and training of the government affairs large language model. S53. The paragraph data obtained from S4 is used as evidence, and the question is pieced together to form a prompt. S54. Input the prompt into the pre-trained large language model obtained in S52, use the reasoning and generation capabilities of the large language model to generate an answer, and return it to the user.

6. The government affairs intelligent question-answering method according to claim 5, characterized in that, S32 includes: dividing the files in the question-and-answer dataset according to prefecture-level cities, obtaining the titles of the documents, and clustering the titles of the documents using the Ward clustering method to obtain similar documents; for the similar documents, selecting those with diversity; using the BERT model trained in S31, vectorizing the questions, obtaining the similarity matrix between questions based on cosine similarity, and then using hierarchical clustering again to obtain questions with high similarity. The similarity acquisition method is as follows: in, and For the question text, This refers to the BERT model that encodes the user's query in DPR retrieval. [CLS] is the start character for the text encoding in the BERT model, and the denominator contains... It is the vector norm; Based on the clustering problem, samples are taken from each category of the problem, and the corresponding documents are obtained. These corresponding documents are then used as the actual training dataset.

Images