Government affair question and answer method and product based on knowledge enhanced large model

Abstract

The application provides a government affair question and answer method and product based on knowledge enhanced large language model, and relates to the technical field of artificial intelligence. The current government affair question and answer system mainly adopts the mode of manual reply, which is low in efficiency, and the system based on RAG lacks query discrimination mechanism, cannot identify intention difference and utilize dialogue context, and affects the quality of retrieved knowledge. In view of the defects existing in the related technologies, the application provides a government affair question and answer method based on knowledge enhanced large model, and specifically, in view of the problem of low efficiency of manual reply, a complete government hotline service framework containing intelligent problem shunting and structured feedback is designed; in view of the problem of poor quality of retrieved knowledge of the traditional RAG system, a multi-stage semantic enhancement retrieval method is designed, and cross-domain knowledge is supplemented through network query, so that accurate response and service closed loop of government consultation are realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a government affairs question-answering method and product based on a knowledge-enhanced large model. Background Technology

[0002] With the development of large-scale language models, the field of intelligent question answering has undergone a significant transformation. Intelligent question answering systems based on large-scale language models have made great progress, especially in vertical fields such as finance and law. Intelligent question answering systems such as FinGPT and Chatlaw have promoted the practical application of the technology and provided strong support for the intelligent processing of professional knowledge.

[0003] As an important bridge for communication between the government and citizens, the government service hotline is becoming increasingly important. Faced with the growing number of inquiries, the existing government Q&A system has certain limitations in dealing with a large number of similar questions, enabling real-time inquiries and efficient transfer. Therefore, it is particularly urgent to build a professional government Q&A system to meet the complex and ever-changing needs. Summary of the Invention

[0004] This invention provides a government affairs question-answering method and product based on a knowledge-enhanced large model, which at least partially solves the above-mentioned problems.

[0005] The first aspect of this invention provides a government affairs question-answering method based on a knowledge-enhanced large model, the method comprising:

[0006] Receive first text information in natural language form sent by the user;

[0007] Based on the historical text information sent by the user and the first text information, a query statement corresponding to the first text information is obtained, and the query statement represents the user's true intention.

[0008] The first N knowledge fragments are retrieved from the government knowledge base based on the query statement;

[0009] The semantic matching degree of the first N knowledge fragments and the query statement is sorted to obtain the first K knowledge fragments with the highest semantic matching degree;

[0010] The usability of the top K knowledge fragments with the highest semantic matching degree is evaluated;

[0011] Introduce usable knowledge fragments as external knowledge into the large language model;

[0012] The answer text for the first text information is obtained based on the endogenous knowledge of the large language model and the external knowledge.

[0013] Optionally, the semantic matching degree of the first N knowledge fragments and the query statement is sorted to obtain the first K knowledge fragments with the highest semantic matching degree, including:

[0014] For any knowledge fragment among the top K knowledge fragments with the highest semantic matching degree:

[0015] The query statement and the knowledge fragment are input as a whole into a large language model. The query and the document are then subjected to deep semantic interaction in conjunction with prompt words. The semantic matching degree between the query statement and the knowledge fragment is then output.

[0016] Optionally, the method further includes:

[0017] If none of the top K semantically most matched knowledge fragments are available, an online query tool is triggered to obtain supplementary knowledge.

[0018] The answer text for the first text information is obtained based on the endogenous knowledge of the large language model and the supplementary knowledge.

[0019] Optionally, the method further includes:

[0020] The intent of the first text information is identified by a large language model to determine the text category corresponding to the first text information.

[0021] If the first text information belongs to the consultation category, external knowledge is retrieved from the government knowledge base based on the query statement and introduced into the large language model. Based on the endogenous knowledge of the large language model and the external knowledge, the answer text for the first text information is obtained.

[0022] If the first text information belongs to the categories of complaint, request for help, or praise, the first text information is summarized and recorded based on the large language model, and an answer text is generated for the first text information.

[0023] If the first text information belongs to another category, an answer text is generated based on the large language model for the first text information.

[0024] Optionally, the method further includes:

[0025] Collect government data from multiple sources;

[0026] The multi-source government data is processed to obtain processed government data;

[0027] The processed government data is processed based on a text encoder model to obtain vectorized data, which is then added to the government knowledge base.

[0028] Optionally, the method further includes:

[0029] A service ticket is generated based on the first text information and the answer text of the first text information using a large language model, and a service ticket verification information is sent based on the service ticket.

[0030] If the service order is verified, the service order is processed by a text encoder model to obtain vectorized data, which is then added to the government knowledge base.

[0031] A second aspect of the present invention provides a government affairs question-answering device based on a knowledge-enhanced big data model, the device comprising:

[0032] The receiving module is used to receive the first text information in natural language form sent by the user;

[0033] The intent analysis module is used to obtain a query statement corresponding to the first text information based on the historical text information sent by the user and the first text information, wherein the query statement represents the user's true intent.

[0034] The retrieval module is used to retrieve the first N knowledge fragments from the government knowledge base based on the query statement; and to sort the semantic matching degree between the first N knowledge fragments and the query statement to obtain the first K knowledge fragments with the highest semantic matching degree.

[0035] An evaluation module is used to evaluate the usability of the top K knowledge fragments with the highest semantic matching degree;

[0036] The answer module is used to introduce usable knowledge fragments as external knowledge into the large language model; and to obtain the answer text for the first text information based on the endogenous knowledge of the large language model and the external knowledge.

[0037] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the government affairs question-answering method based on a knowledge-enhanced big data model as described in the first aspect of the present invention.

[0038] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the government affairs question-answering method based on a knowledge-enhanced big data model as described in the first aspect of the present invention.

[0039] The fifth aspect of the present invention provides a computer program product, including a computer program / instructions, which are implemented by a processor as the steps in the government affairs question-answering method based on a knowledge-enhanced big data model as described in the first aspect of the present invention.

[0040] Current government inquiry systems primarily rely on manual responses, which is inefficient. Furthermore, RAG-based systems lack query discrimination mechanisms, failing to identify differences in intent and utilize dialogue context, thus impacting the quality of retrieved knowledge. To address these shortcomings, this invention proposes a government inquiry method based on a knowledge-enhanced large-scale model. Specifically, to address the inefficiency of manual responses, a complete government hotline service framework incorporating intelligent question triage and structured feedback is designed. To address the poor quality of retrieved knowledge in traditional RAG systems, a multi-stage semantic enhancement retrieval method is designed, supplemented with cross-domain knowledge through online queries, achieving accurate responses and a closed-loop service for government inquiries.

[0041] Comparative experimental results show that the technical solution provided by the embodiments of the present invention improves human evaluation by 27.3% compared with commercial dialogue models, and improves answer recall by 35.6% compared with the fine-tuning model (GLM4-9b-chat). In terms of retrieval quality, compared with the naive RAG system, the technical solution provided by the embodiments of the present invention improves query-knowledge relevance and knowledge support by 15% and 24.6% respectively. In terms of overall performance in multi-turn dialogues, the technical solution provided by the embodiments of the present invention improves the core indicators such as professionalism and accuracy by an average of more than 30% compared with the naive RAG system (based on GLM4-9b-chat), the fine-tuning model (GLM4-9b-chat), and the commercial large model. Attached Figure Description

[0042] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of the steps of the government affairs question-answering method based on a knowledge-enhanced large model provided by the present invention;

[0044] Figure 2 This is a flowchart illustrating the stage-based semantically enhanced knowledge retrieval process of the government affairs question-answering method based on a knowledge-enhanced large model provided by the present invention.

[0045] Figure 3 This is a flowchart illustrating the response triage mechanism of the government affairs question-answering method based on a knowledge-enhanced big data model provided by this invention.

[0046] Figure 4 This is a schematic diagram of the framework of the government affairs question-answering system based on a knowledge-enhanced big data model provided by the present invention;

[0047] Figure 5This is a structural block diagram of the government affairs question-answering device based on a knowledge-enhanced big data model provided by the present invention. Detailed Implementation

[0048] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] Compared to other professional fields, large-scale model applications in government affairs scenarios face unique challenges: the ambiguity of policy texts, the strong standardization of service processes, and the high frequency of knowledge updates make it difficult for existing general-purpose models to meet the needs. Government hotlines often involve policy interpretation and real-time updates of public information across multiple departments. Directly using LLM (Learning by Model) may result in "illusions" in processing these tasks. While knowledge graph-based question-answering systems can achieve accurate matching through the core design of structured knowledge bases, they face limitations such as high construction costs and slow dynamic updates. In contrast, the Retrieval-Augmented Generation (RAG) framework, by integrating unstructured text retrieval and generation technologies, significantly expands the information coverage while maintaining professionalism. A typical RAG system consists of three main modules: data processing, knowledge base construction and querying, and answer generation. By introducing external professional knowledge, it effectively enhances the application capabilities of large-scale models in vertical domains. However, existing RAG systems suffer from two key problems in practical applications: first, they lack discriminatory processing of user input, indiscriminately retrieving knowledge from the knowledge base for each query; second, queries rely solely on the current user input, failing to effectively filter redundant information that may affect query accuracy and easily overlooking crucial information from the dialogue history. These problems result in less than ideal quality of retrieved knowledge, sometimes even negatively impacting answer generation.

[0050] Based on this, this invention proposes a government affairs question-answering method based on a knowledge-enhanced large-scale model. Addressing the problems of low efficiency in manual responses and difficulties in integrating multi-source policy information in the government affairs field, this method innovatively applies large-scale model technology to government affairs, proposing and designing a new government affairs question-answering system based on a knowledge-enhanced large-scale model. Specifically, addressing the problems of low-quality text backpropagation and information omission in multi-turn dialogues existing question-answering systems, this invention introduces a multi-stage semantically enhanced knowledge retrieval method to improve retrieval accuracy during the knowledge base construction and query stages, and introduces online query as a cross-domain knowledge supplementation means, thereby providing more accurate and comprehensive professional knowledge. In the answer generation stage, an intent recognition and question triage model based on the large-scale model is designed, which can automatically identify the question type—questions requiring professional knowledge support will trigger the knowledge retrieval enhancement module, while questions that the model can handle independently will directly generate answers. This design reduces the illusion that external knowledge introduction may bring and improves the system's response efficiency. Furthermore, for the needs of government hotline scenarios, a summary feedback module has been added, which uses intelligent information extraction to structure the dialogue content, achieving a service loop and providing data support for continuous optimization.

[0051] Specifically, such as Figure 1 The diagram illustrates a flowchart of a government affairs question-answering method based on a knowledge-enhanced large model, provided by an embodiment of the present invention. The method includes the following steps:

[0052] S101 receives the first text information in natural language form sent by the user.

[0053] The government affairs question-and-answer method provided in this embodiment of the invention is applied to a government affairs question-and-answer system, which can provide a user interface to receive first text information in natural language form sent by users.

[0054] In this embodiment of the invention, the first text information in natural language form is natural language text sent by the user, expressing the user's government affairs Q&A needs.

[0055] S102, based on the historical text information sent by the user and the first text information, a query statement corresponding to the first text information is obtained, and the query statement represents the user's true intention.

[0056] S103, based on the query statement, retrieve the first N knowledge fragments from the government knowledge base; sort the semantic matching degree between the first N knowledge fragments and the query statement to obtain the first K knowledge fragments with the highest semantic matching degree.

[0057] S104, evaluate the availability of the top K knowledge fragments with the highest semantic matching degree.

[0058] In this embodiment of the invention, steps S102-S104 constitute a multi-stage semantically enhanced knowledge retrieval process provided in this embodiment. This process can solve the problems of low-quality text backpropagation and information omission in multi-turn dialogues in related technologies. Specifically, as shown... Figure 2 As shown, it illustrates a flowchart of a multi-stage semantically enhanced knowledge retrieval process in an embodiment of the present invention.

[0059] In this embodiment of the invention, after receiving the current dialogue (i.e., the first text information) sent by the user, the historical text information sent by the user can be further considered to obtain the query statement corresponding to the first text information, specifically as follows: Figure 2 The historical dialogue summary retrieval stage is shown in the figure.

[0060] In this embodiment of the invention, historical text information refers to the historical conversations sent by the current user before sending the current conversation.

[0061] Traditional sparse text retrieval techniques based on word frequency statistics (such as BM25 and TF-IDF) mainly rely on word frequency for retrieval, but their fault tolerance is weak when handling synonyms and near-synonyms. While content-based dense retrieval techniques reduce information loss through complete Transformer inference, their high time cost makes them unsuitable for large-scale retrieval. Although vector-based dense retrieval techniques optimize computational efficiency, compressing documents into a single vector inevitably leads to information loss. Therefore, to achieve a balance between query accuracy and time cost, this invention proposes a multi-stage semantically enhanced knowledge retrieval process for government knowledge base retrieval, such as... Figure 2 As shown, it includes three core stages: the historical dialogue summary and retrieval stage, the semantic reordering stage, and the self-reflection and decision-making stage.

[0062] Specifically, key entities can be extracted from historical dialogues, the user's true intent can be extracted from the current dialogue, and the key entities can be used as supplementary information to rewrite the user's input based on the user's true intent (e.g., ...). Figure 2 The historical dialogue summary retrieval stage yields a query statement, which can be used to more accurately match content in the knowledge base.

[0063] In this embodiment of the invention, a rewriting model can be constructed based on a large language model plus prompt words to identify and remove redundant information in user input, extract core semantics and key intents, and transform complex, colloquial expressions into standardized search statements. Specifically, in terms of multi-turn dialogue enhancement, by analyzing historical dialogues, key information omitted in user input is recovered, pronouns and referents are parsed, and a coherent and complete query semantic representation is constructed, laying a solid foundation for subsequent accurate retrieval of the knowledge base, thereby improving the accuracy and relevance of the retrieval. This rewriting process can be represented by the following formula:

[0064] (1)

[0065] in, This is the rewritten query statement. LLM Representing a large language model, This is a prompt word template specifically for rewriting tasks. It is the first text information. H It is historical text information.

[0066] Furthermore, in this embodiment of the invention, the query statement can be retrieved from the government knowledge base.

[0067] In this embodiment of the invention, the retrieval step is implemented based on step S103 (e.g. Figure 2 (Semantic reordering stage in the process). In this embodiment of the invention, a multi-stage semantically enhanced knowledge retrieval is adopted in the retrieval step. First, the query statement is encoded to obtain a query vector, and N semantic vectors are retrieved from the government knowledge base based on the query vector. Specifically, the rewritten user query is first processed... Vectorization to obtain the query q Subsequently, a vector-based dense retrieval technique was used to quickly retrieve the first N knowledge fragments from the government knowledge base. (like Figure 2 The knowledge index (1~N) in the semantic reordering stage. This process can be expressed by the following formula:

[0068] (2)

[0069] Where D represents a knowledge fragment in the government knowledge base, and Distance is the distance function.

[0070] In this embodiment of the invention, a more complex semantic matching algorithm is further introduced to perform in-depth contextual analysis and fine evaluation on the preliminary retrieval results (the top N knowledge fragments) and to select the top-K knowledge fragments with the best semantic matching and the richest information.

[0071] In this embodiment of the invention, the semantic matching degree of the first N knowledge fragments and the query statement is sorted to obtain the first K knowledge fragments with the highest semantic matching degree, including:

[0072] For any knowledge fragment among the top K knowledge fragments with the highest semantic matching degree:

[0073] The query statement and the knowledge fragment are input as a whole into a large language model. The query and the document are then subjected to deep semantic interaction in conjunction with prompt words. The semantic matching degree between the query statement and the knowledge fragment is then output.

[0074] In this embodiment of the invention, a cross-encoder model can also be used to perform deep semantic interaction between the query and the document through a self-attention mechanism, and output the semantic matching degree between the query statement and the knowledge fragment.

[0075] Specifically, in this embodiment of the invention, the selection of the first N knowledge fragments can be expressed as the following formula:

[0076] (3)

[0077] in This represents a cross-encoder model that takes a query q and the knowledge fragments d from the top N knowledge fragments as a whole as input. Using a Transformer architecture, it performs deep semantic interaction between the query and the document through a self-attention mechanism, outputting a score between 0 and 1, representing the semantic matching degree between the query and the document. The TopK() function sorts all document-query pairs output by the CrossEncoder and selects the K documents with the highest scores as the final result. K is an adjustable hyperparameter, typically much smaller than the candidate set size N in the first stage.

[0078] In this embodiment of the invention, the aforementioned semantic reordering stage achieves multi-dimensional filtering of knowledge fragments. However, considering the capacity limitations of the government knowledge base, its coverage still has certain limitations. For queries that exceed the coverage of the knowledge base, even if the most relevant knowledge fragments are obtained through the first two stages, these retrieval results may still not be sufficient to support the response to the original query, and may even lead to misleading outputs from large-scale language models, causing negative impacts. Therefore, this embodiment of the invention further proposes to conduct usability assessment on the retrieved knowledge fragments.

[0079] Specifically, such as Figure 2 As shown in the self-reflection decision-making stage, in this embodiment of the invention, after obtaining the search results, a self-reflection evaluation module is constructed through prompt word engineering. This module determines the actual value of knowledge fragments based on a large language model. The evaluation module assigns a binary label to each knowledge fragment. Its evaluation function is:

[0080] (4)

[0081] Where C represents the knowledge fragments retrieved from the knowledge base (i.e., the top K knowledge fragments with the highest semantic matching degree). Based on the evaluation results, the following decision-making mechanism is adopted for knowledge selection in this embodiment of the invention:

[0082] (5)

[0083] S105, usable knowledge fragments are introduced into the large language model as external knowledge; based on the endogenous knowledge of the large language model and the external knowledge, the answer text for the first text information is obtained.

[0084] In this embodiment of the invention, if it is determined that the top K knowledge fragments with the highest semantic matching degree are usable, the knowledge fragments are injected as external knowledge into the large language model so that the large language model can generate answer text for the first text information to reply to the current dialogue sent by the user.

[0085] In one optional implementation, the method further includes:

[0086] If none of the top K semantically most relevant knowledge fragments are available, an online query tool is triggered to obtain supplementary knowledge; based on the endogenous knowledge of the large language model and the supplementary knowledge, an answer text for the first text information is obtained.

[0087] In this embodiment of the invention, when the matching degree of the internal knowledge base does not meet the threshold requirement, the external tool can be automatically triggered to call the interface Tool() to realize the dynamic supplementation of cross-domain knowledge, thereby improving the overall response quality of the system.

[0088] In one optional implementation, the method further includes:

[0089] S1, perform intent recognition on the first text information using a large language model to determine the text category corresponding to the first text information.

[0090] S2, if the first text information belongs to the consultation category, external knowledge is retrieved from the government knowledge base based on the query statement and introduced into the large language model, and an answer text for the first text information is obtained based on the endogenous knowledge of the large language model and the external knowledge; if the first text information belongs to the complaint, request for help, or praise category, the first text information is summarized and recorded based on the large language model, and an answer text for the first text information is generated; if the first text information belongs to other categories, an answer text for the first text information is generated based on the large language model.

[0091] In this embodiment of the invention, the user's first text information is classified using an intent recognition model based on prompt word engineering and a large language model:

[0092] (6)

[0093] in These correspond to different types of questions: inquiries, complaints / suggestions, and others. For complaints / suggestions and other questions, the large language model will respond according to its assigned roles and workflows, and based on its own capabilities.

[0094] For consultation-type questions, the system uses a multi-stage semantically enhanced knowledge retrieval method to determine whether cross-domain knowledge enhancement is needed: →{0,1}, where =1 indicates that knowledge support is required. When knowledge support is needed, the online query tool Tool is triggered to retrieve supplementary knowledge.

[0095] (7)

[0096] Finally, the system integrates the original query q and the final knowledge and inputs it into the large language model to generate the answer:

[0097] (8)

[0098] Specifically, such as Figure 3 The diagram illustrates a flowchart of the response triage mechanism provided in this embodiment of the invention. In this embodiment, the intent recognition model based on a large language model can classify the first text information sent by the user. If the first text information is classified as an inquiry, a knowledge fragment can be obtained based on a multi-stage semantically enhanced knowledge retrieval process. The availability of the knowledge fragment determines whether an online query is needed to obtain supplementary knowledge. The retrieved knowledge fragment is then used as external knowledge, or the supplementary knowledge obtained from the online query is injected into the large language model to provide a knowledge-based response and generate a reply. If the first text information is classified as other categories, the large language model can autonomously respond to the first text information based on prompt words, generating a reply. If the first text information is classified as a complaint, request for help, or praise, the large language model summarizes and records the first text information based on prompt words, generating a reply.

[0099] This invention proposes to classify and triage user dialogues. This innovative triage mechanism and dynamic knowledge supplementation strategy not only avoid redundant retrieval of simple questions, but also make up for the deficiencies of the knowledge base through online queries, significantly improving the system's answer quality and knowledge coverage.

[0100] In one optional implementation, the method further includes:

[0101] S11 collects multi-source government data.

[0102] S12, perform data processing on the multi-source government data to obtain processed government data.

[0103] S13, The processed government data is processed based on the text encoder model to obtain vectorized data, and the vectorized data is added to the government knowledge base.

[0104] In this embodiment of the invention, data such as Q&A data from the 12345 government service platform, information from government agencies in various provinces and cities, and local laws and regulations can be collected as multi-source government data.

[0105] Furthermore, in this embodiment of the invention, multi-source government data can be processed. Specifically, the multi-source government data can be parsed and sliced ​​to obtain processed government data, which may include question-and-answer blocks and text blocks. These question-and-answer blocks and text blocks can be further processed to obtain vectorized data, and further to obtain a government knowledge base.

[0106] In this embodiment of the invention, the government knowledge base can serve as an offline database to provide external knowledge for subsequent retrieval.

[0107] In one optional implementation, the method further includes:

[0108] S21, a service work order is generated based on the first text information and the answer text of the first text information using a large language model, and work order verification information is sent based on the service work order.

[0109] S22, if the service work order is verified, the service work order is processed by a text encoder model to obtain vectorized data, and the vectorized data is added to the government knowledge base.

[0110] In this embodiment of the invention, a large language model can summarize the answer text obtained from the first text information of the current user based on the prompt word engineering to generate a service work order. The service work order may include: event title, event content, and processing result, etc., wherein the event title and event content can be obtained from the summary of the first text information, and the processing result can be obtained from the summary of the answer text.

[0111] In this embodiment of the invention, after a service work order is generated, work order verification information can be sent to the corresponding verification terminal. The verification terminal confirms whether the service work order contains errors. If the service work order passes verification, the content corresponding to the service work order can be added to the government knowledge base as a new knowledge fragment. Specifically, the service work order can be processed using a text encoder model to obtain vectorized data, which is then added to the government knowledge base.

[0112] In this embodiment of the invention, the completed government affairs question-and-answer process is summarized by a large language model to obtain new knowledge fragments, which are then added to the government affairs knowledge base, allowing for continuous replenishment of the government affairs knowledge base.

[0113] In this invention, addressing the issues of low efficiency in manual responses and difficulties in integrating multi-source policy information in the government affairs sector, a novel large-scale model technology is innovatively applied to government affairs, proposing and designing a new government affairs question-answering method based on a knowledge-enhanced large-scale model. Furthermore, to address the problems of low-quality text backpropagation and information omission in multi-turn dialogues existing in current question-answering systems, a multi-stage semantically enhanced knowledge retrieval method is designed. Finally, an automatic evaluation framework for question-answering systems based on large-scale models and prompt word engineering is constructed, enabling a comprehensive evaluation of system performance.

[0114] Based on the same inventive concept, this invention also provides a government affairs question-answering system based on a knowledge-enhanced large-scale model, such as... Figure 4 As shown, it illustrates the framework of a government affairs question-answering system based on a knowledge-enhanced big data model. This system includes: a data processing module, a knowledge base construction and query module, an answer generation module, and a summary and feedback module.

[0115] The data processing module collects multi-source government data, including Q&A data from the 12345 government service platform, information from provincial and municipal government agencies, and local laws and regulations. This multi-source data is then parsed and segmented to obtain Q&A blocks and text blocks. These blocks are further processed to produce vectorized data, which is then used to create a government knowledge base. This knowledge base is then used as an offline database to provide external knowledge for subsequent retrieval.

[0116] The knowledge base construction and query module is used to introduce a multi-stage semantically enhanced knowledge retrieval process and to introduce online query as a cross-domain knowledge supplementation method, thereby providing more accurate and comprehensive professional knowledge. The specific process is similar to the above steps S102~S104, and will not be repeated here.

[0117] The answer generation module is used to receive the first text information sent by the user and generate answer text according to the corresponding response strategy based on the question category. The specific process is similar to steps S1~S2 above, and will not be repeated here.

[0118] The summary and feedback module is used to summarize the answer text generated by the current large language model and combine it with the corresponding first text information to obtain a service ticket. Based on the verified service ticket, the government knowledge base is supplemented. The specific process is similar to the above steps S21~S22, and will not be repeated here.

[0119] This invention also provides a comparative verification embodiment, which is experimentally verified on the Ubuntu 22.04 Linux operating system. The hardware configuration is as follows: Processor: 12th Gen Intel(R) Core(TM) i7-12700; Graphics Processor: NVIDIA GeForce RTX 4090 (24GB VRAM); System Memory: 1024GB. Based on a comprehensive consideration of the model's performance in Chinese tasks, inference efficiency, and security, this invention uses GLM-9b-chat developed by Zhipu AI as the basic language model.

[0120] Based on the documents listed in Table 1, this embodiment of the invention constructs a test dataset for government knowledge Q&A.

[0121] Table 1. Knowledge Base Document Statistics

[0122]

[0123] The dataset includes two main categories: single-turn question-and-answer dialogues and multi-turn question-and-answer dialogues. The single-turn dialogue dataset is further divided into data not yet included in the database and data from the knowledge base. The multi-turn dialogue dataset comprises a multi-turn dialogue dataset rewritten from question-and-answer data from the 12345 government service platform. Specific test dataset information is shown in Table 2.

[0124] Table 2. Statistics on Dataset Issues

[0125]

[0126] Traditional RAG evaluation methods face a triple dilemma: semantic quantification bias, evaluation cost, and result reliability. Rule-based metrics (such as ROUGE-L) rely on text overlap measurement performance and struggle to capture semantic consistency; human annotation frameworks (such as TruLens) offer high reliability but are expensive; pure LLM automated evaluation (such as RAGAS) is low-cost but differs significantly from human evaluation. Therefore, this invention combines the authority of human annotation with the scalability of LLM evaluation, and incorporates the Chain of Thought (CoT) method to deconstruct the evaluation object into independent units for fine-grained analysis. By calculating the overall performance through the average unit scores, a comprehensive and systematic question-answering system evaluation method is constructed.

[0127] The evaluation framework focuses on two core dimensions: retrieval quality and generation quality, employing a semantic relevance quantitative scoring mechanism ranging from 0 to 10. The evaluation objects are defined as: Q: query; C: retrieval knowledge; A: system answer; T: actual answer.

[0128] The retrieval quality assessment includes three core dimensions: 1) Query-knowledge relevance (QC_relevance), which assesses the degree of support by calculating the semantic matching degree between the retrieved knowledge fragments and the query; 2) True answer-knowledge relevance (TC_relevance), which analyzes the coverage of retrieved knowledge with standard answers; and 3) Knowledge support, which detects the knowledge support of each sentence in the generated answer to verify factuality and reliability.

[0129] (9)

[0130] (10)

[0131] (11)

[0132] Where K, M, and N represent the total number of retrieved knowledge fragments, the total number of sentences with the actual answer T, and the total number of sentences with the system's answer A, respectively. A large model of prompt words representing the semantic similarity between evaluation objects.

[0133] The quality assessment focuses on two dimensions: 1) Answer similarity (TA_similarity), which quantifies the semantic coverage of the system's answer and the standard answer; 2) Human evaluation (User_acceptance), which measures user satisfaction from dimensions such as ease of understanding and interactive experience.

[0134] (12)

[0135] It is a large-scale prompt word model that calculates the semantic similarity between the true answer T and the system answer A.

[0136] This multi-dimensional, fine-grained evaluation method provides a comprehensive and objective performance evaluation framework for the RAG question answering system, aiming to accurately quantify the degree of semantic association between evaluation objects.

[0137] This invention provides a comprehensive evaluation of the performance of the ChatGovt system through rigorous comparative experiments. We selected three representative benchmark systems: (1) a naive RAG system based on GLM4-9b-chat, which uses only the dense vector retrieval method for knowledge acquisition; (2) an enhanced model obtained by fine-tuning GLM4-9b-chat using the constructed knowledge base data under 4-bit quantization conditions via the LoRA method; and (3) a commercial large model with online query capabilities, as a reference standard for the current technical level.

[0138] Based on a carefully constructed single-turn dialogue dataset and a continuous follow-up questioning scenario, a test set was built. The system's generation performance was systematically evaluated from two key dimensions: retrieval quality and generation quality. The evaluation results are shown in Table 3.

[0139] Table 3 System Response Evaluation Results

[0140]

[0141] As can be seen, in terms of the quality of generated answers, the government affairs question-answering system based on the knowledge-enhanced large model provided in this embodiment of the invention achieves excellent scores of 9.19 points in human evaluation and 6.34 points in answer recall, with all generation quality indicators significantly outperforming the benchmark system. Specifically, its answer recall is 55.4% higher than the fine-tuned model, and its human evaluation score exceeds that of the commercial large model by 27.3%, demonstrating the effectiveness of the external knowledge base. Comparative analysis reveals that although the fine-tuned model outperforms the naive RAG system (3.08 points) in answer recall (4.08 points), its human evaluation score (5.72 points) is lower than the latter (7.97 points). This may be due to the overemphasis on the mechanical reproduction of policy texts during the fine-tuning process, resulting in generated results that, while conforming to standard expressions, lack natural dialogue connectors and contextual explanations. This suggests that relying solely on model fine-tuning may reduce the practicality of the answers due to overfitting.

[0142] In terms of knowledge retrieval quality, the government affairs question-answering system based on the knowledge-enhanced big data model demonstrates significant technical advantages. As shown in Table 3, its three core indicators—query-knowledge relevance (7.20), actual answer-knowledge relevance (7.00), and knowledge support (7.25)—are all superior to the naive RAG system, improving by 15.0%, 7.4%, and 24.6%, respectively. This improvement stems from the effective integration of hybrid retrieval strategies: through the synergistic effect of multi-stage semantically enhanced knowledge retrieval methods, the system can capture deep semantic connections and accurately locate professional terms in policy texts, thus achieving "dual assurance" of knowledge location in complex government affairs scenarios.

[0143] Based on the same inventive concept, this invention also provides a government affairs question-answering device based on a knowledge-enhanced large model, such as... Figure 5 As shown, a structural block diagram of a government affairs question-answering device based on a knowledge-enhanced big data model is illustrated. The government affairs question-answering device 500 based on the knowledge-enhanced big data model includes:

[0144] The receiving module 501 is used to receive first text information in natural language form sent by the user;

[0145] The intent analysis module 502 is used to obtain a query statement corresponding to the first text information based on the historical text information sent by the user and the first text information, wherein the query statement represents the user's true intent.

[0146] The retrieval module 503 is used to retrieve the first N knowledge fragments from the government knowledge base based on the query statement; and to sort the semantic matching degree between the first N knowledge fragments and the query statement to obtain the first K knowledge fragments with the highest semantic matching degree.

[0147] Evaluation module 504 is used to evaluate the usability of the top K knowledge fragments with the highest semantic matching degree;

[0148] The answer module 505 is used to introduce usable knowledge fragments as external knowledge into the large language model; and to obtain the answer text for the first text information based on the endogenous knowledge of the large language model and the external knowledge.

[0149] Optionally, the semantic matching degree of the first N knowledge fragments and the query statement is sorted to obtain the first K knowledge fragments with the highest semantic matching degree, including:

[0150] For any knowledge fragment among the top K knowledge fragments with the highest semantic matching degree:

[0151] The query statement and the knowledge fragment are input as a whole into a large language model. The query and the document are then subjected to deep semantic interaction in conjunction with prompt words. The semantic matching degree between the query statement and the knowledge fragment is then output.

[0152] Optionally, the device further includes:

[0153] The online query module is used to trigger the online query tool to obtain supplementary knowledge when none of the top K knowledge fragments with the highest semantic matching degree are available;

[0154] The answer module is also used to obtain the answer text for the first text information based on the endogenous knowledge of the large language model and the supplementary knowledge.

[0155] Optionally, the device further includes:

[0156] The classification module is used to perform intent recognition on the first text information using a large language model, and to determine the text category corresponding to the first text information.

[0157] The retrieval module 503 is used to retrieve external knowledge from the government knowledge base based on the query statement when the first text information belongs to the consultation category, and to introduce it into the large language model, and to obtain the answer text for the first text information based on the endogenous knowledge of the large language model and the external knowledge.

[0158] The answer module 505 is also used to summarize and record the first text information based on a large language model and generate answer text for the first text information when the first text information belongs to the categories of complaint, request for help, and praise.

[0159] The answer module 505 is further configured to generate an answer text for the first text information based on a large language model when the first text information belongs to another category.

[0160] Optionally, the device further includes:

[0161] The collection module is used to collect government data from multiple sources;

[0162] The data processing module is used to process the multi-source government data to obtain processed government data;

[0163] The knowledge base construction module is used to process the processed government data based on the text encoder model to obtain vectorized data, and then add the vectorized data to the government knowledge base.

[0164] Optionally, the device further includes:

[0165] The summary module is used to generate a service ticket based on the first text information and the answer text of the first text information using a large language model, and to send the service ticket verification information based on the service ticket.

[0166] The feedback module is used to process the service work order through a text encoder model to obtain vectorized data when the service work order is verified, and then add the vectorized data to the government knowledge base.

[0167] Based on the same inventive concept, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of the government affairs question-answering method based on a knowledge-enhanced large language model as described in any of the above embodiments.

[0168] Based on the same inventive concept, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the government affairs question-answering method based on a knowledge-enhanced large language model described in any of the above embodiments.

[0169] Based on the same inventive concept, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the government affairs question-answering method based on a knowledge-enhanced large language model described in any of the above embodiments.

[0170] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0171] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] This invention is described with reference to flowchart illustrations and / or block diagrams of the method, terminal device (apparatus), and computer program product according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable terminal device, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0173] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable terminal device to cause a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0175] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0176] Finally, it should be noted that in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0177] The above provides a detailed description of a government affairs question-answering method based on a knowledge-enhanced large language model provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A government affair question and answer method based on a knowledge enhanced large model, characterized in that, The method comprises: receiving first text information in natural language form sent by a user; based on historical text information sent by the user and the first text information, obtaining a query statement corresponding to the first text information, the query statement representing the real intention of the user; based on the query statement, retrieving N pieces of knowledge fragments from a government knowledge base; sorting the semantic matching degrees of the N pieces of knowledge fragments and the query statement to obtain K pieces of knowledge fragments with the highest semantic matching degrees; constructing a self-reflection evaluation module through a prompt word engineering, evaluating the availability of the K pieces of knowledge fragments with the highest semantic matching degrees based on a first large language model to obtain an evaluation result; in the case that the evaluation result is that the K pieces of knowledge fragments with the highest semantic matching degrees are available, introducing the available knowledge fragments as external knowledge into a second large language model, and obtaining an answer text for the first text information based on the endogenous knowledge of the second large language model and the external knowledge; in the case that the evaluation result is that the K pieces of knowledge fragments with the highest semantic matching degrees are all unavailable, triggering a network query tool to obtain supplementary knowledge, and obtaining an answer text for the first text information based on the endogenous knowledge of the second large language model and the supplementary knowledge.

2. The government affair question answering method based on a knowledge enhanced large model according to claim 1, characterized in that, The method further comprises: performing intention recognition on the first text information through a large language model to determine a text classification corresponding to the first text information; in the case that the first text information belongs to the consultation category, introducing external knowledge into the large language model based on the query statement from the government knowledge base, and obtaining an answer text for the first text information based on the endogenous knowledge of the large language model and the external knowledge; 3. The government affair question answering method based on a knowledge enhanced large model according to claim 1, characterized in that, in the case that the first text information belongs to the complaint category, the help-seeking category, and the praise category, summarizing and recording the first text information based on the large language model, and generating an answer text for the first text information; in the case that the first text information belongs to other categories, generating an answer text for the first text information based on the large language model. The method further comprises: collecting multi-source government data; performing data processing on the multi-source government data to obtain processed government data; 4. The government affair question answering method based on a knowledge-enhanced large model according to any one of claims 1-3, characterized in that, processing the processed government data based on a text encoder model to obtain vectorized data, and adding the vectorized data to the government knowledge base. The method further comprises: generating a service work order based on the first text information and the answer text for the first text information through a large language model, and sending work order verification information based on the service work order; ​ 5. The government affair question answering method based on a knowledge enhanced large model according to claim 4, characterized in that, ​ ​ In the case that the service order verification passes, the service order is processed by a text encoder model to obtain vectorized data, and the vectorized data is supplemented into a government knowledge base.

6. A government question answering device based on a knowledge enhanced large model, the device comprising: a receiving module configured to receive first text information in a natural language form sent by a user; an intent analysis module configured to obtain a query statement corresponding to the first text information based on historical text information sent by the user and the first text information, the query statement representing a real intent of the user; a retrieving module configured to retrieve N pieces of knowledge from a government knowledge base based on the query statement; sort the semantic matching degrees of the N pieces of knowledge and the query statement to obtain K pieces of knowledge with the highest semantic matching degrees; an evaluation module configured to construct a self-reflection evaluation module through prompt word engineering, evaluate the availability of the K pieces of knowledge with the highest semantic matching degrees based on a first large language model, and obtain an evaluation result; a answering module configured to, in the case that the evaluation result indicates that the K pieces of knowledge have availability, introduce the knowledge with availability as external knowledge into a second large language model, obtain an answer text for the first text information based on endogenous knowledge of the second large language model and the external knowledge, and in the case that the evaluation result indicates that none of the K pieces of knowledge has availability, trigger a network query tool to obtain supplementary knowledge, and obtain an answer text for the first text information based on endogenous knowledge of the second large language model and the supplementary knowledge.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the government question answering method based on the knowledge enhanced large model according to any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the government question answering method based on the knowledge enhanced large model according to any one of claims 1-5.

9. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to realize the steps in the government question answering method based on the knowledge enhanced large model according to any one of claims 1-5.

Images