Government affair intelligent question and answer method and system based on pre-training model enhancement
By constructing a government knowledge graph and a multi-feature fusion question classification model, the accuracy problems of entity recognition and question parsing in the government question-and-answer system were solved, and efficient government question-and-answer services were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUJIAN TERTON SOFTWARE CO LTD
- Filing Date
- 2026-04-27
- Publication Date
- 2026-07-10
AI Technical Summary
Existing government question-and-answer systems suffer from low entity recognition accuracy, large relationship extraction errors, inaccurate question intent recognition, and insufficient ability to parse complex questions, making it difficult to meet the intelligent needs of government services.
We employ a pre-trained model enhancement approach, constructing a government affairs knowledge graph using the PW2NER+CASREL entity relationship joint extraction model, combining the SBERT+ATT+Biaffine question classification model for question parsing, and utilizing the Neo4j graph database for accurate retrieval and answer generation.
It significantly improved the accuracy of government entity identification and relationship extraction, enhanced the accuracy of question intent recognition, and improved the intelligence level and user experience of the government question-and-answer system.
Smart Images
Figure CN122364394A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of natural language processing and knowledge graph application technology, specifically relating to a government affairs intelligent question answering method and system based on pre-trained model enhancement. Background Technology
[0002] With the digital transformation of government information and the popularization of artificial intelligence applications, government departments have increasingly higher demands for knowledge-based intelligent question-and-answer applications. Government question-and-answer systems have become an important vehicle for improving the efficiency of government services and optimizing user experience. Knowledge graphs, as a structured knowledge representation framework, provide solid knowledge support for government question-and-answer systems. However, existing systems still face many problems in practical applications: First, government texts often exhibit overlapping and discontinuous entity characteristics, leading to low accuracy in traditional entity extraction models, which are difficult to adapt to the characteristics of government texts. Second, the pipelined approach to entity relation extraction is prone to error propagation, resulting in poor triple extraction and affecting the quality of the knowledge graph. Third, government questions often contain mixed interrogative words, numerous entity synonyms, complex modifiers, and mismatched entity references, making traditional question classification models insufficient in intent recognition accuracy and limiting their ability to parse complex questions. Fourth, existing government question-and-answer systems struggle to effectively combine with knowledge graphs to achieve accurate answer retrieval and generation, and the question-and-answer effectiveness and intelligence level need improvement.
[0003] To address the aforementioned issues, there is an urgent need to construct a knowledge graph adapted to the characteristics of the government affairs sector and design high-precision question parsing and question-answering methods to improve the overall performance of the government affairs question-answering system and meet the development needs of digital and intelligent government services. Summary of the Invention
[0004] The purpose of this invention is to propose a government affairs intelligent question answering method and system based on pre-trained model enhancement, in order to solve the problems of low entity recognition accuracy, large relationship extraction error, inaccurate question intent recognition, and insufficient ability to parse complex government affairs questions in existing government affairs question answering systems. It can effectively improve the accuracy of government affairs entity relationship extraction and question intent recognition.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: a government affairs intelligent question answering method based on pre-trained model enhancement, comprising the following steps:
[0006] Step S1: Standardize the original government documents data, extract the government entity relationship triples using the PW2NER+CASREL-based entity relationship joint extraction model, and construct a government knowledge graph stored in the graph database.
[0007] Step S2: For the government affairs natural language questions input by the user, through entity extraction, question classification, and question-answer enhancement processing, the core entities and user intent in the questions are analyzed, and the unstructured questions are converted into structured query triples that can be recognized by the graph database.
[0008] Step S3: Convert the structured query triples into graph database query statements, perform accurate retrieval in the government knowledge graph, format and optimize the retrieval results, and then provide the user with the final answer and related visualization information to realize government Q&A.
[0009] Furthermore, step S1 specifically includes:
[0010] Step S101: Preprocess the government text data, including removing special symbols, converting between traditional and simplified characters, completing and standardizing the time and location, and segmenting sentences to obtain the preprocessed effective government data;
[0011] Step S102: Use the Stanza natural language processing tool to perform part-of-speech tagging on government texts, construct the Stanza-POS part-of-speech feature matrix, integrate the part-of-speech feature matrix into the convolutional input layer of the W2NER model, and replace the original encoding layer of the model with a MACBERT encoding layer adapted to Chinese semantics to obtain the optimized PW2NER entity extraction model and complete the model training.
[0012] Step S103: Integrate the PW2NER entity extraction model into the head entity recognition module of the CASREL model, replace the original head entity recognition layer of CASREL, and perform multi-task learning by fusing the PW2NER head entity recognition loss and the final loss of the CASREL model to construct the PW2NER+CASREL entity relationship joint extraction model; use the PW2NER+CASREL entity relationship joint extraction model to process effective government data, automatically identify the head entity, tail entity and semantic relationship between entities in the text, and extract them to form government triplet groups of entity-relationship-object;
[0013] Step S104: Design the ontology of government knowledge graph, define the entity categories, relationship types and attribute information in the government domain, organize the government triples according to the government domain ontology specification, import the triple data into the Neo4j graph database through Cypher query statement, complete the construction of government knowledge graph, and realize the visualization of entities and relationships in the knowledge graph.
[0014] Further, in step S102, the PW2NER entity extraction model includes a MACBERT encoding layer, a Stanza-POS part-of-speech input layer, a matrix convolutional layer, and a prediction layer, which extracts word pairs from government documents. The relationship score is calculated as follows:
[0015]
[0016] in, For word pairs High-order feature representation extracted by dilated convolutional layers For the weight vector, For bias terms;
[0017] The loss function of the PW2NER entity extraction model is the binary cross-entropy loss function, calculated as follows:
[0018]
[0019] in, The length of the input sequence; Indicator pairs The true label; The Sigmoid function maps the score to... Interval; Normalization term Used to eliminate the influence of the length of the input sequence on the dimension of the loss.
[0020] Further, in step S103, the calculation formula for the CASREL model is as follows:
[0021]
[0022] Where x represents the input sentence, Let s represent a triple, where s is the head entity, r is the relation, and o is the tail entity. This means that given an input sentence x, the model predicts the triples. The probability of; This represents a chain multiplication operation; T represents the set of all head entities in the input sentence x. This represents the probability that the model predicts for the head entity s given an input sentence x; This represents the set of relations containing all triples of the tail entity o; This represents the probability that, given an input sentence x and a head entity s, the model predicts the relation r and the tail entity o. Represents the set of relations containing all triples of the head entity s; This represents the probability that the model predicts the tail entity o given an input sentence x, a head entity s, and a relation r; This indicates excluding from the predefined full set of relations R. The set of relations remaining after the set; This represents the probability that, given an input sentence x, a head entity s, and a relation r, the model predicts that the tail entity is "empty".
[0023] Furthermore, step S2 specifically includes:
[0024] Step S201: Receive the government affairs natural language question input by the user, clean the text and input it into the PW2NER entity extraction model to complete the candidate entity extraction and obtain the candidate entity set in the government affairs question, which serves as the candidate head entity for knowledge graph query.
[0025] Step S202: Construct an SBERT+ATT+Biaffine question classification model. Use the SBERT model to perform global semantic encoding on government affairs questions to obtain the semantic vector representation of the questions. Use BERT to encode candidate entities and preset keywords in the government affairs domain to obtain the semantic vector representation of entities and keywords.
[0026] Step S203: Use cross-attention mechanism to fuse question semantic features, candidate entity features and keyword features, construct the distance matrix between candidate entities and keywords and perform dilated convolution processing to extract multi-scale local features;
[0027] Step S204: Input the fused features into the Biaffine classifier to complete the question type classification. According to the preset mapping rules between question types and knowledge graph relationships, convert the classification results into corresponding semantic relationships in the knowledge graph.
[0028] Step S205: Construct a vector index for the government knowledge base using FAISS, implement entity linking using vector index nearest neighbor search, and then supplement the parent entity by performing syntactic dependency analysis using the spacy model to complete question answering enhancement.
[0029] Step S206: Integrate candidate entities, semantic relationships obtained from question classification mapping, entity correction results after question-answering enhancement processing, and parent entity association relationships to select core entities and relationships that conform to the knowledge graph query specifications, and generate structured query triples that can be directly retrieved from the government knowledge graph.
[0030] Furthermore, in step S202, the SBERT model performs global semantic encoding on government affairs questions by performing average pooling on the vectors output by BERT and then using cosine similarity for differential comparison. The calculation formula is as follows:
[0031]
[0032] in, Cosine similarity, representing the cosine value of the angle between vectors u and v; u i v represents the i-th component of the first vector u; i Let represent the i-th component of the second vector v; n represents the total dimension of the vector.
[0033] Furthermore, in step S203, when fusing features using the cross-attention mechanism, entity embeddings are used as queries, and keyword embeddings are used as keys and values. The calculation formula is as follows:
[0034]
[0035]
[0036]
[0037] in, For entity embedding, Embed keywords; The weight matrix is a learnable weight matrix; These represent the query vector, key vector, and value vector, respectively.
[0038] Attention weight The calculation formula is as follows:
[0039]
[0040] in, The dimension of the key vector; and the dimension of the value vector. Multiplication generates an entity embedding representation that incorporates keyword information. And weighted fusion of sentence embeddings:
[0041]
[0042] in, To integrate weights, S represents the sentence embedding;
[0043] For dilated convolution, let the distance matrix between the input entity keywords be denoted as . ,in Let represent the number of candidate entities, the number of keywords, and the embedding dimension, respectively. Then, for the void ratio... The convolution of is calculated using the following formula:
[0044]
[0045]
[0046] in, These are the kernel weights; This is a bias term.
[0047] Furthermore, in step S204, the question relationship is classified using the Biaffine classifier, and its calculation formula is as follows:
[0048]
[0049] in, The fused feature vector; for The One element; The distance matrix is the first... Line number The elements of the column represent keywords. With entity The distance between them; The weight tensor obtained through learning; This is a bias term.
[0050] Furthermore, in step S205, the entity link calculates the similarity between the candidate entity and the knowledge base entity using Euclidean distance to achieve nearest neighbor matching. The calculation formula is as follows:
[0051]
[0052]
[0053] in, Let be the vector representation of the candidate entity. A set of vectors representing entities in the knowledge base. For vector dimensions.
[0054] Furthermore, step S3 specifically includes:
[0055] Step S301: Based on the generated structured query triples, automatically construct the corresponding Cypher query statement according to the Cypher syntax rules of the Neo4j graph database, send the query statement to the Neo4j graph database, perform a retrieval operation in the government knowledge graph, and obtain the initial answer data matching the question, including relevant entity, relation and attribute information;
[0056] Step S302: Based on the scenario requirements of government affairs Q&A, preset answer templates corresponding to different question types, substitute the initial answer data obtained from knowledge graph retrieval into the matching answer template, complete the structured formatting of the answer, and make the answer expression conform to the question-and-answer habits of natural language;
[0057] Step S303: Call the large language model API and guide the model to optimize the answer by using preset prompts, including simplifying, organizing and beautifying the answer. At the same time, perform factual verification and timeline consistency verification on the answer, generate the final government affairs question and answer answer and display the visualization results of the government affairs knowledge graph subgraph.
[0058] Compared with the prior art, the present invention has the following beneficial effects:
[0059] 1. This invention addresses the characteristics of overlapping and discontinuous entities in government documents by improving the W2NER model to obtain the PW2NER entity extraction model. It introduces part-of-speech matrix to enhance semantic capture capabilities and replaces the MACBERT encoding layer to adapt to Chinese government documents, effectively improving the accuracy of government document entity recognition. The F1 score of the PW2NER entity extraction model is 1.23% higher than that of the baseline model W2NER.
[0060] 2. This invention constructs a PW2NER+CASREL joint entity relationship extraction model, integrating PW2NER into the head entity recognition module of CASREL and performing loss fusion, which alleviates the error propagation problem of pipelined methods. On the Baidu dataset and government data set, the F1 score is improved by 1.77% and 2.80% respectively compared with the benchmark model CASREL, achieving accurate extraction of entity relationship triples from government texts, laying the foundation for building a high-quality government knowledge graph.
[0061] 3. This invention constructs a multi-feature fusion question classification model based on SBERT+ATT+Biaffine, which integrates the features of sentences, candidate entities, and keywords and performs convolution processing. The F1 score of question classification is 1.88% higher than the benchmark model SBERT, and the accuracy reaches 92.5%, effectively improving the accuracy of intent recognition for government affairs questions.
[0062] 4. This invention designs a question-answering enhancement method based on entity linking and syntactic dependency analysis, which solves the problems of mismatched entity references and complex attributive components in government affairs questions. This increases the number of triples that the question-answering system can extract by 21.9%, improves the parsing ability for complex government affairs questions, and ensures the accuracy of answer retrieval.
[0063] 5. This invention achieves intelligent question answering based on government knowledge graphs. By combining preset answer templates with answer optimization using a large language model, it not only improves the accuracy and readability of question answering results, but also detects and prompts errors in query results. It also supports extended queries, which greatly improves the intelligence level and user experience of the government question answering system. It can be widely applied to digital and intelligent government service scenarios. Attached Figure Description
[0064] Figure 1 This is a flowchart illustrating the implementation of the government affairs intelligent question-answering method based on pre-trained model enhancement provided in this embodiment of the invention.
[0065] Figure 2 This is a schematic diagram of the structure of the PW2NER entity extraction model in an embodiment of the present invention;
[0066] Figure 3 This is a schematic diagram of the structure of the PW2NER+CASREL entity relationship joint extraction model in an embodiment of the present invention;
[0067] Figure 4 This is a schematic diagram of the structure of the SBERT+ATT+Biaffine question classification model in an embodiment of the present invention;
[0068] Figure 5 This is an overall architecture diagram of the government affairs intelligent question-and-answer system in this embodiment of the invention. Detailed Implementation
[0069] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0070] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.
[0071] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0072] like Figure 1 As shown in the figure, this embodiment provides a government affairs intelligent question answering method based on pre-trained model enhancement, and its implementation steps are as follows.
[0073] Step S1: Standardize the original government documents data, extract the government entity relationship triples using the PW2NER+CASREL-based entity relationship joint extraction model, and construct a government knowledge graph stored in the graph database.
[0074] Step S101: Preprocess the government affairs text data, including removing special symbols, converting between traditional and simplified characters, completing and standardizing the time and location, and segmenting sentences to obtain the preprocessed valid government affairs data.
[0075] Step S102: Use the Stanza natural language processing tool to perform part-of-speech tagging on government documents, construct the Stanza-POS part-of-speech feature matrix, integrate the part-of-speech feature matrix into the convolutional input layer of the W2NER model, and replace the original encoding layer of the model with a MACBERT encoding layer adapted to Chinese semantics to obtain the optimized PW2NER entity extraction model, thus completing the model training. The structure of the PW2NER entity extraction model is as follows: Figure 2 As shown.
[0076] The PW2NER entity extraction model includes a MACBERT encoding layer, a Stanza-POS part-of-speech input layer, a matrix convolutional layer, and a prediction layer. It is used for word pair extraction in government texts. The relationship score is calculated as follows:
[0077]
[0078] in, For word pairs High-order feature representation extracted by dilated convolutional layers For the weight vector, This is a bias term.
[0079] The loss function of the PW2NER entity extraction model is the binary cross-entropy loss function, calculated as follows:
[0080]
[0081] in, The length of the input sequence; Indicator pairs The true label; The Sigmoid function maps the score to... Interval; Normalization term Used to eliminate the influence of the length of the input sequence on the dimension of the loss.
[0082] Step S103: Integrate the PW2NER entity extraction model into the head entity recognition module of the CASREL model, replacing the original CASREL head entity recognition layer. Combine the PW2NER head entity recognition loss with the CASREL model's final loss for multi-task learning to construct a PW2NER+CASREL joint entity relationship extraction model, as follows: Figure 3 As shown, the PW2NER+CASREL entity-relationship joint extraction model is used to process effective government data, automatically identifying head entities, tail entities, and semantic relationships between entities in the text, and extracting them into entity-relationship-object government triples.
[0083] The calculation formula for the CASREL model is as follows:
[0084]
[0085] Where x represents the input sentence, Let s represent a triple, where s is the head entity, r is the relation, and o is the tail entity. This means that given an input sentence x, the model predicts the triples. The probability of; This represents a chain multiplication operation; T represents the set of all head entities in the input sentence x. This represents the probability that the model predicts for the head entity s given an input sentence x; This represents the set of relations containing all triples of the tail entity o; This represents the probability that, given an input sentence x and a head entity s, the model predicts the relation r and the tail entity o. Represents the set of relations containing all triples of the head entity s; This represents the probability that the model predicts the tail entity o given an input sentence x, a head entity s, and a relation r; This indicates excluding from the predefined full set of relations R. The set of relations remaining after the set; This represents the probability that, given an input sentence x, a head entity s, and a relation r, the model predicts that the tail entity is "empty".
[0086] Step S104: Design the ontology of government knowledge graph, define the entity categories, relationship types and attribute information in the government domain, organize the government triples according to the government domain ontology specification, import the triple data into the Neo4j graph database through Cypher query statement, complete the construction of government knowledge graph, and realize the visualization of entities and relationships in the knowledge graph.
[0087] Step S2: For the government affairs natural language questions input by the user, through entity extraction, question classification, and question-answer enhancement processing, the core entities and user intent in the questions are analyzed, and the unstructured questions are converted into structured query triples that can be recognized by the graph database.
[0088] Step S201: Receive the government affairs natural language question input by the user, clean the text and input it into the PW2NER entity extraction model to complete the candidate entity extraction and obtain the candidate entity set in the government affairs question, which will be used as the candidate head entity for knowledge graph query.
[0089] Step S202: Construct an SBERT+ATT+Biaffine question classification model. The SBERT model performs global semantic encoding on government affairs questions to obtain semantic vector representations of the questions. BERT is used to encode candidate entities and pre-defined keywords in the government affairs domain to obtain semantic vector representations of the entities and keywords. The structure of the SBERT+ATT+Biaffine question classification model is as follows: Figure 4 As shown.
[0090] The SBERT model performs global semantic encoding on government affairs questions by average pooling the vectors output by BERT and then using cosine similarity for difference comparison. The calculation formula is as follows:
[0091]
[0092] in, Cosine similarity, representing the cosine value of the angle between vectors u and v; u i v represents the i-th component of the first vector u; i Let represent the i-th component of the second vector v; n represents the total dimension of the vector.
[0093] Step S203: Use cross-attention mechanism to fuse question semantic features, candidate entity features and keyword features, construct the distance matrix between candidate entities and keywords and perform dilated convolution processing to extract multi-scale local features.
[0094] When fusing features using the cross-attention mechanism, entity embeddings are used as queries, and keyword embeddings are used as keys and values. The calculation formula is as follows:
[0095]
[0096]
[0097]
[0098] in, For entity embedding, Embed keywords; The weight matrix is a learnable weight matrix; These represent the query vector, key vector, and value vector, respectively.
[0099] Attention weight The calculation formula is as follows:
[0100]
[0101] in, The dimension of the key vector; and the dimension of the value vector. Multiplication generates an entity embedding representation that incorporates keyword information. And weighted fusion of sentence embeddings:
[0102]
[0103] in, For weight fusion, S is the sentence embedding.
[0104] For dilated convolution, let the distance matrix between the input entity keywords be denoted as . ,in Let represent the number of candidate entities, the number of keywords, and the embedding dimension, respectively. Then, for the void ratio... The convolution of is calculated using the following formula:
[0105]
[0106]
[0107] in, These are the kernel weights; This is a bias term.
[0108] Step S204: Input the fused features into the Biaffine classifier to complete the question type classification. According to the preset mapping rules between question types and knowledge graph relationships, convert the classification results into corresponding semantic relationships in the knowledge graph.
[0109] The Biaffine classifier is used to classify question relationships, and its calculation formula is as follows:
[0110]
[0111] in, The fused feature vector; for The One element; The distance matrix is the first... Line number The elements of the column represent keywords. With entity The distance between them; The weight tensor obtained through learning; This is a bias term.
[0112] Step S205: Construct a vector index for the government knowledge base using FAISS, implement entity linking using vector index nearest neighbor search, and then supplement the parent entity through syntactic dependency analysis using the spacy model to complete question answering enhancement.
[0113] The entity links are calculated using Euclidean distance to determine the similarity between candidate entities and entities in the knowledge base, achieving nearest neighbor matching. The calculation formula is as follows:
[0114]
[0115]
[0116] in, Let be the vector representation of the candidate entity. A set of vectors representing entities in the knowledge base. For vector dimensions.
[0117] Step S206: Integrate candidate entities, semantic relationships obtained from question classification mapping, entity correction results after question-answering enhancement processing, and parent entity association relationships to select core entities and relationships that conform to the knowledge graph query specifications, and generate structured query triples that can be directly retrieved from the government knowledge graph.
[0118] Step S3: Convert the structured query triples into graph database query statements, perform accurate retrieval in the government knowledge graph, format and optimize the retrieval results, and then provide the user with the final answer and related visualization information to realize government Q&A.
[0119] Step S301: Based on the generated structured query triples, automatically construct the corresponding Cypher query statement according to the Cypher syntax rules of the Neo4j graph database, send the query statement to the Neo4j graph database, perform a retrieval operation in the government knowledge graph, and obtain the initial answer data matching the question, including relevant entity, relation and attribute information.
[0120] Step S302: Based on the scenario requirements of government affairs Q&A, preset answer templates corresponding to different question types, substitute the initial answer data obtained from knowledge graph retrieval into the matching answer template, complete the structured formatting of the answer, and make the answer expression conform to the question-and-answer habits of natural language.
[0121] Step S303: Call the large language model API and guide the model to optimize the answer by using preset prompts, including simplifying, organizing and beautifying the answer. At the same time, perform factual verification and timeline consistency verification on the answer, generate the final government affairs question and answer answer and display the visualization results of the government affairs knowledge graph subgraph.
[0122] This embodiment also provides a government affairs intelligent question-answering system based on a pre-trained model, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method. The specific architecture of the government affairs intelligent question-answering system is as follows: Figure 5 As shown.
[0123] This embodiment also provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method.
[0124] Experimental verification
[0125] I. Experimental Environment
[0126] In this embodiment, the experimental hardware environment uses a Linux operating system, equipped with an Intel(R) Xeon(R) CPU E5-2620v4@2.10GHz processor, a Tesla P100-PCIE graphics card, 16GB of RAM, and a 512GB hard drive. The software environment is built on Python 3.8.13, using frameworks such as Torch 1.10.0, Stanza 1.4.2 / 1.5.0, and Transformers 4.13.0. The graph database used is Neo4j community version -2025.01.0. The model training hyperparameters are set to MAX_LEN=300, Epoch=50 / 100, Batch_size=64, and Learning_rate=0.00001 / 0.0001. The evaluation metrics used are precision, recall, and their harmonic mean F1 score, which are common in information extraction and text classification. Precision is used as an auxiliary evaluation metric for the question classification task.
[0127] II. Experimental Dataset
[0128] 1. For the entity relation extraction dataset, it is divided into the Baidu public dataset and the government-specific dataset. The Baidu dataset contains about 58,000 Chinese sentences, covering various types of text such as news and encyclopedia, with 35% overlapping entities. The government dataset was crawled from the public information of the Fujian Provincial Government Portal using the Scrapy framework. After preprocessing, 13,390 valid data were obtained, including core government entities such as meetings, policies, institutions, and people. It has characteristics such as discontinuous entities, overlap, and complex syntactic components. It is divided into training set, validation set, and test set in a 7:2:1 ratio.
[0129] 2. The question classification dataset consists of a mixture of manually annotated government affairs questions and the THUCNews dataset, each accounting for 50%. Ten common government affairs question types are predefined and mapped one-to-one with knowledge graph relationships, including policy inquiries, meeting inquiries, exhibition inquiries, etc., covering the core scenarios of government affairs question answering.
[0130] 3. For the question-answering effect verification dataset, 3,000 valid questions were extracted from the question classification data to verify the question-answering effect of the question-answering enhancement module and the overall system.
[0131] III. Comparative Experimental Design
[0132] To verify the innovation and effectiveness of the proposed method, multiple comparative experiments were designed, targeting the four core aspects of entity extraction, joint entity relation extraction, question classification, and question-answering enhancement. The model in this invention was compared with current mainstream benchmark models / methods in the field. The specific experimental setup is as follows:
[0133] 1. Entity extraction experiment: Using W2NER as the baseline model, compare it with models such as Hypergraph-based and BERT-BiLSTM-CRF to verify the performance improvement of the PW2NER entity extraction model in government entity recognition.
[0134] 2. Joint entity relationship extraction experiment: Using CASREL as the benchmark model, compare it with models such as Bert+BiLSTM+CRF (pipeline method) and SpERT to verify the effectiveness of the PW2NER+CASREL joint entity relationship extraction model on Baidu dataset and government dataset.
[0135] 3. Question classification experiment: Using SBERT as the baseline model, the classification accuracy of the SBERT+ATT+Biaffine question classification model is verified by comparing it with classic classification models such as TextCNN and ERNIE-CNN.
[0136] IV. Experimental Results and Analysis
[0137] 1. Entity Extraction Experiment Results and Analysis
[0138] Table 1 shows the experimental results of each entity extraction model on the government data set. The PW2NER entity extraction model proposed in this invention achieves a precision of 76.66%, a recall of 76.23%, and an F1 score of 76.43%, which is 1.23% higher than the baseline model W2NER and 2.23% and 5.26% higher than BERT-BiLSTM-CRF and Hypergraph-based models, respectively.
[0139] Table 1 Experimental results of the entity extraction model
[0140]
[0141] Ablation experiments further validated the role of each improved module in the PW2NER entity extraction model. Removing the Stanza-POS part-of-speech matrix decreased the model's F1 score by 1.19%; restoring the MACBERT encoding layer to BERT resulted in a 0.39% decrease in the F1 score. Experimental results demonstrate that this invention, by introducing the Stanza-POS part-of-speech feature matrix to enhance semantic capture capabilities and replacing the MACBERT encoding layer with semantic features adapted to Chinese government documents, effectively solves the problems of discontinuous and overlapping entities in government documents, significantly improving the accuracy of entity extraction. This proves the targetedness and effectiveness of the PW2NER entity extraction model improvement strategy.
[0142] 2. Results and Analysis of Joint Extraction of Entity Relationships
[0143] Table 2 shows the experimental results of the joint entity relationship extraction model on the Baidu dataset and the government dataset. The PW2NER+CASREL joint entity relationship extraction model proposed in this invention achieves an F1 score of 82.17% on the Baidu dataset and 80.15% on the government dataset. Compared with the benchmark model CASREL, the F1 scores on the two datasets are improved by 1.77% and 2.80% respectively, which is significantly better than the Bert+BiLSTM-CRF pipeline model and the SpERT model.
[0144] Table 2 Experimental Results of the Joint Entity Relationship Extraction Model
[0145]
[0146] Ablation experiments verified the effectiveness of the core improvements in the model. Removing the weighted loss fusion from the head and tail recognition modules decreased the F1 score by 0.65%; replacing PW2NER with a regular linear layer decreased the F1 score by 1.71%. Experimental results demonstrate that this invention integrates PW2NER into the head entity recognition module of CASREL, achieving multi-task learning through loss fusion. This effectively alleviates the error propagation problem of traditional pipelined methods. Furthermore, the entity recognition process is optimized for the domain characteristics of government data sets, resulting in superior performance in entity relation extraction from complex government texts. This proves the innovation and adaptability of the PW2NER+CASREL joint entity relation extraction model.
[0147] 3. Results and Analysis of the Question Classification Experiment
[0148] Table 3 shows the experimental results of each question classification model on a mixed dataset. The SBERT+ATT+Biaffine question classification model proposed in this invention achieves a precision of 92.20%, a recall of 92.54%, an F1 score of 92.50%, and an accuracy of 92.5%. Compared with the baseline model SBERT, the F1 score is improved by 1.88%, and compared with ERNIE-CNN and TextCNN models, the improvements are 1.83% and 2.70%, respectively.
[0149] Table 3 Experimental results of the question classification model
[0150]
[0151] Experimental results show that this invention integrates multi-dimensional features of questions, candidate entities, and keywords through a cross-attention mechanism, combines dilated convolution to process local features of the entity-keyword distance matrix, and finally achieves accurate classification through the Biaffine classifier. This effectively solves the problems of mixed interrogative words and many entity synonyms in government affairs questions, improves the accuracy of question intent recognition, and proves the effectiveness of the multi-feature fusion question classification model.
[0152] Through multiple sets of comparative and ablation experiments, the innovation and effectiveness of this invention and its core technical modules were verified:
[0153] 1. In response to the domain characteristics of discontinuous and overlapping entities in government documents, the PW2NER entity extraction model proposed in this invention effectively improves the entity recognition accuracy by introducing a part-of-speech feature matrix and replacing the MACBERT encoding layer, thus laying a high-quality data foundation for knowledge graph construction.
[0154] 2. The PW2NER+CASREL entity relationship joint extraction model constructed in this invention achieves multi-task learning by fusing head entity recognition loss, which alleviates the error propagation problem and has a more significant performance improvement on government data sets, adapting to the complex features of government texts.
[0155] 3. The SBERT+ATT+Biaffine multi-feature fusion question classification model proposed in this invention effectively integrates the semantic features of questions, entities, and keywords, improves the accuracy of intent recognition for government affairs questions, and provides technical support for accurately analyzing user needs;
[0156] The core technology modules of this invention work together to form an intelligent question-and-answer solution adapted to the characteristics of the government affairs field, providing a feasible technical path for the digital and intelligent transformation of government services.
[0157] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0158] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0159] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0160] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0161] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.
Claims
1. A government affairs intelligent question-answering method based on pre-trained model enhancement, characterized in that, Includes the following steps: Step S1: Standardize the original government documents data, extract the government entity relationship triples using the PW2NER+CASREL-based entity relationship joint extraction model, and construct a government knowledge graph stored in the graph database. Step S2: For the government affairs natural language questions input by the user, through entity extraction, question classification, and question-answer enhancement processing, the core entities and user intent in the questions are analyzed, and the unstructured questions are converted into structured query triples that can be recognized by the graph database. Step S3: Convert the structured query triples into graph database query statements, perform accurate retrieval in the government knowledge graph, format and optimize the retrieval results, and then provide the user with the final answer and related visualization information to realize government Q&A.
2. The government affairs intelligent question answering method based on pre-trained model enhancement according to claim 1, characterized in that, Step S1 specifically includes: Step S101: Preprocess the government text data, including removing special symbols, converting between traditional and simplified characters, completing and standardizing the time and location, and segmenting sentences to obtain the preprocessed effective government data; Step S102: Use the Stanza natural language processing tool to perform part-of-speech tagging on government texts, construct the Stanza-POS part-of-speech feature matrix, integrate the part-of-speech feature matrix into the convolutional input layer of the W2NER model, and replace the original encoding layer of the model with a MACBERT encoding layer adapted to Chinese semantics to obtain the optimized PW2NER entity extraction model and complete the model training. Step S103: Integrate the PW2NER entity extraction model into the head entity recognition module of the CASREL model, replace the original head entity recognition layer of CASREL, and perform multi-task learning by fusing the PW2NER head entity recognition loss and the final loss of the CASREL model to construct the PW2NER+CASREL entity relationship joint extraction model; use the PW2NER+CASREL entity relationship joint extraction model to process effective government data, automatically identify the head entity, tail entity and semantic relationship between entities in the text, and extract them to form government triplet groups of entity-relationship-object; Step S104: Design the ontology of government knowledge graph, define the entity categories, relationship types and attribute information in the government domain, organize the government triples according to the government domain ontology specification, import the triple data into the Neo4j graph database through Cypher query statement, complete the construction of government knowledge graph, and realize the visualization of entities and relationships in the knowledge graph.
3. The government affairs intelligent question answering method based on pre-trained model enhancement according to claim 2, characterized in that, In step S102, the PW2NER entity extraction model includes a MACBERT encoding layer, a Stanza-POS part-of-speech input layer, a matrix convolutional layer, and a prediction layer, which extracts word pairs from government documents. The relationship score is calculated as follows: in, For word pairs High-order feature representation extracted by dilated convolutional layers For the weight vector, For bias terms; The loss function of the PW2NER entity extraction model is the binary cross-entropy loss function, calculated as follows: in, The length of the input sequence; Indicator pairs The true label; The Sigmoid function maps the score to... Interval; Normalization term Used to eliminate the influence of the length of the input sequence on the dimension of the loss.
4. The government affairs intelligent question answering method based on pre-trained model enhancement according to claim 2, characterized in that, In step S103, the calculation formula for the CASREL model is as follows: Where x represents the input sentence, Let s represent a triple, where s is the head entity, r is the relation, and o is the tail entity. This means that given an input sentence x, the model predicts the triples. The probability of; This represents a chain multiplication operation; T represents the set of all head entities in the input sentence x; This represents the probability that the model predicts for the head entity s given an input sentence x; This represents the set of relations containing all triples of the tail entity o; This represents the probability that, given an input sentence x and a head entity s, the model predicts the relation r and the tail entity o. This represents the set of relations containing all triples of the head entity s; This represents the probability that the model predicts the tail entity o given an input sentence x, a head entity s, and a relation r. This indicates excluding from the predefined full set of relations R. The set of relations remaining after the set; This represents the probability that, given an input sentence x, a head entity s, and a relation r, the model predicts that the tail entity is "empty".
5. The government affairs intelligent question answering method based on pre-trained model enhancement according to claim 1, characterized in that, Step S2 specifically includes: Step S201: Receive the government affairs natural language question input by the user, clean the text and input it into the PW2NER entity extraction model to complete the candidate entity extraction and obtain the candidate entity set in the government affairs question, which serves as the candidate head entity for knowledge graph query. Step S202: Construct an SBERT+ATT+Biaffine question classification model. Use the SBERT model to perform global semantic encoding on government affairs questions to obtain the semantic vector representation of the questions. Use BERT to encode candidate entities and preset keywords in the government affairs domain to obtain the semantic vector representation of entities and keywords. Step S203: Use cross-attention mechanism to fuse question semantic features, candidate entity features and keyword features, construct the distance matrix between candidate entities and keywords and perform dilated convolution processing to extract multi-scale local features; Step S204: Input the fused features into the Biaffine classifier to complete the question type classification. According to the preset mapping rules between question types and knowledge graph relationships, convert the classification results into corresponding semantic relationships in the knowledge graph. Step S205: Construct a vector index for the government knowledge base using FAISS, implement entity linking using vector index nearest neighbor search, and then supplement the parent entity by performing syntactic dependency analysis using the spacy model to complete question answering enhancement. Step S206: Integrate candidate entities, semantic relationships obtained from question classification mapping, entity correction results after question-answering enhancement processing, and parent entity association relationships to select core entities and relationships that conform to the knowledge graph query specifications, and generate structured query triples that can be directly retrieved from the government knowledge graph.
6. The government affairs intelligent question answering method based on pre-trained model enhancement according to claim 5, characterized in that, In step S202, the SBERT model performs global semantic encoding on government affairs questions. This involves average pooling the vectors output by BERT and then using cosine similarity for differential comparison. The calculation formula is as follows: in, Cosine similarity, representing the cosine value of the angle between vectors u and v; u i v represents the i-th component of the first vector u; i Let represent the i-th component of the second vector v; n represents the total dimension of the vector.
7. The government affairs intelligent question answering method based on pre-trained model enhancement according to claim 5, characterized in that, In step S203, when fusing features using the cross-attention mechanism, entity embeddings are used as queries, and keyword embeddings are used as keys and values. The calculation formula is as follows: in, For entity embedding, Embed keywords; The weight matrix is a learnable weight matrix; These represent the query vector, key vector, and value vector, respectively. Attention weight The calculation formula is as follows: in, The dimension of the key vector; and the dimension of the value vector. Multiplication generates an entity embedding representation that incorporates keyword information. And weighted fusion of sentence embeddings: in, To integrate weights, S represents the sentence embedding; For dilated convolution, let the distance matrix between the input entity keywords be denoted as . ,in Let represent the number of candidate entities, the number of keywords, and the embedding dimension, respectively. Then, for the void ratio... The convolution of is calculated using the following formula: in, These are the kernel weights; This is a bias term.
8. The government affairs intelligent question answering method based on pre-trained model enhancement according to claim 5, characterized in that, In step S204, the question relationship is classified using the Biaffine classifier, and the calculation formula is as follows: in, The fused feature vector; for The One element; The distance matrix is the first... Line number The elements of the column represent keywords. With entity The distance between them; The weight tensor obtained through learning; This is a bias term.
9. The government affairs intelligent question answering method based on pre-trained model enhancement according to claim 5, characterized in that, In step S205, the entity link calculates the similarity between the candidate entity and the knowledge base entity using Euclidean distance to achieve nearest neighbor matching. The calculation formula is as follows: in, Let be the vector representation of the candidate entity. A set of vectors representing entities in the knowledge base. For vector dimensions.
10. The government affairs intelligent question answering method based on pre-trained model enhancement according to claim 1, characterized in that, Step S3 specifically includes: Step S301: Based on the generated structured query triples, automatically construct the corresponding Cypher query statement according to the Cypher syntax rules of the Neo4j graph database, send the query statement to the Neo4j graph database, perform a retrieval operation in the government knowledge graph, and obtain the initial answer data matching the question, including relevant entity, relation and attribute information; Step S302: Based on the scenario requirements of government affairs Q&A, preset answer templates corresponding to different question types, substitute the initial answer data obtained from knowledge graph retrieval into the matching answer template, complete the structured formatting of the answer, and make the answer expression conform to the question-and-answer habits of natural language; Step S303: Call the large language model API and guide the model to optimize the answer by using preset prompts, including simplifying, organizing and beautifying the answer. At the same time, perform factual verification and timeline consistency verification on the answer, generate the final government affairs question and answer answer and display the visualization results of the government affairs knowledge graph subgraph.