Government affairs intelligent question answering method, device, equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a government affairs intelligent question-answering method, device, equipment and storage medium. The method uses a large model to enhance the existing data in the FAQ question-answering database, extracts FAQ question-answering pairs and splits document paragraphs for government affairs documents transmitted through an API interface of a government affairs network, uses a Structbert algorithm to retrieve FAQ question-answering pairs of a query request query, and outputs them when the similarity is higher than a preset threshold. Otherwise, a three-branch decision hybrid retrieval of the query request query and each government affairs document, including bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval and dynamic self-attention DynAttenBiLSTM content retrieval, is executed to obtain a hybrid retrieval result. Finally, a government affairs answer large model is used to output the question-answering result of the query request query based on the hybrid retrieval result. Thus, through a hybrid retrieval module integrating three-branch decision modules and multiple similarity algorithms, the document answer can be traced, the answer reliability is strong, and the accuracy is high.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a government affairs intelligent question-answering method, device, equipment and storage medium. Background Art

[0002] The emergence of large language models has brought rapid development to natural language processing technology, especially in document question-answering systems. Traditional question-answering systems and intelligent customer service robots mainly rely on existing question-answer pairs in the database for retrieval, and use document matching, question-answer database matching algorithms combined with text and word positions to find highly similar documents to answer common user questions.

[0003] At present, the intelligent question and answer system of the government service network mainly relies on the response support of existing question and answer data in the traditional sense and manual replies to users' questions. Government service documents and policy documents are not fully utilized and mined. Manual replies are time-consuming and costly. In addition, the query performance of traditional retrieval methods is significantly reduced when dealing with massive data. The generalization performance is highly dependent on database retrieval information, and the recalled answers are difficult to meet users' search needs for precision and high relevance.

[0004] Therefore, how to improve the accuracy and reliability of government questions and answers and achieve traceability of document answers is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a government affairs intelligent question-answering method, device, equipment and storage medium, aiming to solve at least one of the above-mentioned technical problems.

[0006] To achieve the above object, the present invention provides a government affairs intelligent question-answering method, comprising the following steps:

[0007] Use the big model to enhance the existing data in the FAQ database, and extract FAQs and split document paragraphs from government service documents passed in through the government service API interface;

[0008] For the user's query request query, the FAQ question-answer pair database is retrieved using the StructBert algorithm and the target government affairs document with the highest similarity is returned. The corresponding FAQ question-answer pairs under the target government affairs document are retrieved using the StructBert algorithm and the target FAQ question-answer pairs with a similarity higher than a preset threshold are returned.

[0009] When there is no target FAQ question-answer pair with a similarity higher than a preset threshold in the target government affairs document, based on the FAQ question-answer pair and document paragraphs corresponding to each government affairs document, the query request query and each government affairs document are subjected to a three-branch decision hybrid search including bce-embedding-base-V1 vectorized search, BM25-scSentenceTransformer hybrid search and dynamic self-attention DynAttenBiLSTM content search to obtain a hybrid search result;

[0010] The mixed search results are transmitted to a large government affairs answering model trained with government affairs service documents, and the question and answer results of the query request query are obtained by output.

[0011] Optionally, the data in the FAQ database is enhanced using the big model, and the FAQ extraction and document paragraph splitting steps are performed on the government service documents transmitted by the government service network API interface, specifically including:

[0012] For the existing FAQ pairs in the FAQ database, the big model is used to expand the questions in each FAQ pair into several semantically similar query requests, and the questions of each existing FAQ pair are replaced with the corresponding generated query requests with similar semantics and stored in the FAQ database, thus realizing data enhancement of the FAQ database.

[0013] Extract key fixed information of the government service document consisting of acceptance conditions, handling procedures, required materials and basic information from the government service document transmitted through the government service network API interface according to a fixed template format, use the document name of the government service document as a question, use the key fixed information of the document as an answer, and construct a FAQ question-answer pair for the government service document;

[0014] Load the doc type source document of the government service document, use the python-docx library to read and clean the document, split it into individual document paragraphs and indexes according to the secondary titles, remove stop words, segment and vectorize the text, and store it in the vector database.

[0015] Optionally, for a user's query request query, using the StructBert algorithm to retrieve the FAQ question-answer pair database and return the target government affairs service document with the highest similarity, and using the StructBert algorithm to retrieve a number of FAQ question-answer pairs corresponding to the target government affairs service document and return the target FAQ question-answer pairs with a similarity higher than a preset threshold, specifically including:

[0016] The initial Structbert model is trained according to the FAQ question and answer database to obtain the Structbert algorithm model for government affairs intelligent question and answer;

[0017] Using the Structbert algorithm model, a semantic similarity search is performed between the user's query request query and the document name of the government affairs document corresponding to each FAQ question and answer database, and the target government affairs document with the highest similarity is returned;

[0018] The Structbert algorithm model is used to perform semantic similarity retrieval on a plurality of FAQs in a database of a user's query request query and the FAQs corresponding to the target government affairs document, and a target FAQ pair with a similarity higher than a preset threshold is returned.

[0019] Optionally, the initial StructBert model is trained according to the FAQ question and answer database to obtain the StructBert algorithm model steps for government affairs intelligent question and answer, specifically including:

[0020] For each government service document corresponding to the FAQ question-answer pair database, after the document paragraphs are split, the big model is used to generate several paragraph-level question-answer pairs for each document paragraph. At the same time, the big model is used to generate several cross-paragraph question-answer pairs for the entire government service document.

[0021] Prepare a number of paragraph-level question-answer pairs, a number of cross-paragraph question-answer pairs, and a number of FAQ question-answer pairs stored in each FAQ question-answer pair database as sample data, and divide the sample data into a training set, a validation set, and a test set according to an 8:1:1 ratio;

[0022] The training set and validation set are used as the input of the initial StructBert model for training until the model converges, and the test set is used to test and evaluate the trained StructBert algorithm model.

[0023] Optionally, based on the FAQ question-answer pair and document paragraph corresponding to each government affairs document, the query request query and each government affairs document including a three-branch decision hybrid retrieval of bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval and dynamic self-attention DynAttenBiLSTM content retrieval are executed to obtain a hybrid retrieval result step, specifically including:

[0024] Execute the query request query and the bce-embedding-base-V1 vectorized retrieval of each government service document: Calculate the semantic similarity between the query request query and several word vectors corresponding to each document paragraph in the vector database, use bce-embedding-base-V1 to sort the document paragraphs that meet the semantic similarity requirements, and obtain the score vector ;

[0025] Perform BM25-scSentenceTransformer hybrid retrieval of the query request query and each government service document: Use the improved BM25 algorithm to retrieve the correlation between the query request query and the document name of the government service document based on probability, return the government service documents that meet the first correlation requirement, and use the government service documents that meet the correlation requirement to train the scSentenceTransformer semantic model, use the trained scSentenceTransformer semantic model to predict the correlation between the query request query and the document paragraphs of the government service document that meet the second correlation requirement, and return the document paragraphs that meet the third correlation requirement and the corresponding score vector ;

[0026] Execute dynamic self-attention DynAttenBiLSTM content retrieval between query request query and each government service document: Build the DynAttenBiLSTM model in combination with the dynamic self-attention layer mechanism, use dynamic query encoding in the self-attention layer to dynamically calculate the similarity between the query and the document content, and dynamically focus on key paragraphs through position encoding when pre-selecting similarity calculations to adjust the attention weight; then introduce the position encoding between the query request query and the document paragraphs of the government service document, add the position encoding to the original word vector to obtain the enhanced word vector, and then use the enhanced cosine similarity to calculate the similarity between the query request query and the document paragraphs of the government service document, and return the document paragraphs that meet the fourth relevance requirements and the corresponding score vector. ;

[0027] A three-branch decision hybrid retrieval including bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval and dynamic self-attention DynAttenBiLSTM content retrieval of the score vector is performed to obtain the hybrid retrieval result.

[0028] Optionally, the step of using an improved BM25 algorithm to retrieve the correlation between the query request query and the document name of the government affairs document based on probability includes:

[0029] Using the weighted global word frequency statistics method, the inverse document frequency (IDF) is improved based on the word frequency information of several topics in government affairs documents.

[0030] Based on the table types in government affairs documents, different weights are assigned to different parts of the table-type government affairs documents, and the improved term frequency TF is calculated;

[0031] Use decision tree DT combined with grid search to define the best hyperparameters for BM25 and , and combined with structured paragraph information to improve the BM25 calculation formula as follows:

[0032] ;

[0033] in, Indicates the number of terms in the query, Is the first terms, is the inverse document frequency, N is the total number of documents, Indicates that it contains a term The number of documents, Representing terms In the documentation The number of occurrences in represents a collection of terms, Representing terms In the documentation The frequency of terms in Representation Document Length, represents the average length of all documents, and is the adjustment parameter.

[0034] Optionally, a three-branch decision hybrid retrieval step including bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval, and dynamic self-attention DynAttenBiLSTM content retrieval of the score vector is performed, specifically including:

[0035] Score vectors for bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval, and dynamic self-attention DynAttenBiLSTM content retrieval A normalization formula is used to map the scores of different scoring criteria of different retrieval methods to the same scale;

[0036] ;

[0037] Define the weight vector Adjust the impact of each method on the results;

[0038] ;

[0039] ;

[0040] In the formula, Represents the score vector The score vector of the i-th retrieval method in, Represents the score vector The smallest score vector in Represents the score vector The largest score vector in represents the initial weight value, is the correction factor, It is the user's historical conversation information and current The weight correction function calculated by the intent category, represents the set of all intent categories, Indicates the click weight of the paragraph type. and Represents the user's historical government dialogue and the current one judged by the big model Intent category;

[0041] According to the dynamically adjusted weight Calculate the fusion score for the candidate set U;

[0042] ;

[0043] In the formula, , They represent three models: bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval, and dynamic self-attention DynAttenBiLSTM content retrieval.

[0044] According to the fusion score, the candidate set U is added to the three decision classifiers;

[0045] ;

[0046] In the formula, H represents the high confidence set, L represents the low confidence set, and M represents the fuzzy set. represents the high confidence fusion score threshold, represents the low confidence fusion score threshold;

[0047] The fuzzy paragraph set M is sorted and rearranged using user history information, and the first fuzzy paragraph after rearrangement is selected to be added to the high confidence result set H, and the final high confidence paragraph result is output. ;

[0048] The rearranged expression is as follows:

[0049] ;

[0050] is the weighted coefficient of semantic similarity, Indicates the current user's inquiry question Semantic vector of historical government affairs dialogue with users The cosine similarity of .

[0051] In addition, in order to achieve the above-mentioned purpose, the present invention also provides a government affairs intelligent question-answering device, comprising:

[0052] The processing module is used to enhance the existing data in the FAQ database using the big model, and to extract FAQs and split document paragraphs from the government service documents passed in by the government service API interface;

[0053] The first retrieval module is used to retrieve the FAQ question-answer pair database using the StructBert algorithm for the user's query request query and return the target government affairs document with the highest similarity, and retrieve several FAQ question-answer pairs corresponding to the target government affairs document using the StructBert algorithm and return the target FAQ question-answer pairs with a similarity higher than a preset threshold;

[0054] The second retrieval module is used for executing the three-branch decision hybrid retrieval of the query request query and each government service document including bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval and dynamic self-attention DynAttenBiLSTM content retrieval based on the FAQ question-answer pair and document paragraph corresponding to each government service document when there is no target FAQ question-answer pair with a similarity higher than a preset threshold in the target government service document to obtain a hybrid retrieval result;

[0055] The output module is used to transmit the mixed search results to the government affairs answering model trained with government affairs documents, and output the question and answer results of the query request query.

[0056] In addition, in order to achieve the above-mentioned purpose, the present invention also provides a government affairs intelligent question and answer device, which includes: a memory, a processor, and a government affairs intelligent question and answer program stored in the memory and executable on the processor, and when the government affairs intelligent question and answer program is executed by the processor, the steps of the government affairs intelligent question and answer method as described above are implemented.

[0057] In addition, in order to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which a government affairs intelligent question and answer program is stored. When the government affairs intelligent question and answer program is executed by a processor, the steps of the above-mentioned government affairs intelligent question and answer method are implemented.

[0058] The beneficial effects of the present invention are: a method, device, equipment and storage medium for intelligent government affairs question and answer are proposed, by using a large model to enhance the existing data in the FAQ database, and extracting FAQ pairs and splitting document paragraphs for government affairs documents transmitted by the government affairs network API interface, using the Structbert algorithm to retrieve the FAQ pairs of the query request query, and output them when the similarity is higher than a preset threshold, otherwise, executing the query request query and each government affairs document including a three-branch decision hybrid search of bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval and dynamic self-attention DynAttenBiLSTM content retrieval to obtain a hybrid retrieval result, and finally using the government affairs answer large model to output the question and answer result of the query request query based on the hybrid retrieval result, thereby, through the hybrid retrieval module that integrates three decision modules and multiple similarity algorithms, the document answer can be traced, the answer reliability is strong, and the accuracy is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention;

[0060] Figure 2 A schematic diagram of a process flow of an embodiment of the government affairs intelligent question-answering method of the present invention;

[0061] Figure 3 It is a structural block diagram of an intelligent government question-and-answer device in an embodiment of the present invention.

[0062] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings.

[0063] Description of reference numerals:

[0064] 10-processing module; 20-first retrieval module; 30-second retrieval module; 40-output module;

[0065] 1001 - processor; communication bus - 1002; 1003 - user interface; 1004 - network interface; 1005 - memory. DETAILED DESCRIPTION

[0066] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0067] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0068] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0069] like Figure 1 As shown, the device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0070] Those skilled in the art will understand that Figure 1 The structure of the device shown in the figure does not constitute a limitation of the device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0071] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a government affairs intelligent question-answering program.

[0072] exist Figure 1In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the government affairs intelligent question and answer program stored in the memory 1005 and perform the following operations:

[0073] Use the big model to enhance the existing data in the FAQ database, and extract FAQs and split document paragraphs from government service documents passed in through the government service API interface;

[0074] For the user's query request query, the FAQ question-answer pair database is retrieved using the StructBert algorithm and the target government affairs document with the highest similarity is returned. The corresponding FAQ question-answer pairs under the target government affairs document are retrieved using the StructBert algorithm and the target FAQ question-answer pairs with a similarity higher than a preset threshold are returned.

[0075] When there is no target FAQ question-answer pair with a similarity higher than a preset threshold in the target government affairs document, based on the FAQ question-answer pair and document paragraphs corresponding to each government affairs document, the query request query and each government affairs document are subjected to a three-branch decision hybrid search including bce-embedding-base-V1 vectorized search, BM25-scSentenceTransformer hybrid search and dynamic self-attention DynAttenBiLSTM content search to obtain a hybrid search result;

[0076] The mixed search results are transmitted to a large government affairs answering model trained with government affairs service documents, and the question and answer results of the query request query are obtained by output.

[0077] The specific embodiments of the present invention applied to the device are basically the same as the embodiments of the following application of the government affairs intelligent question and answer method, and will not be repeated here.

[0078] The embodiment of the present invention provides a government affairs intelligent question-answering method, referring to Figure 2 , Figure 2 It is a flow chart of an embodiment of the government affairs intelligent question-answering method of the present invention.

[0079] In this embodiment, the government affairs intelligent question answering method includes the following steps:

[0080] S100: Use the big model to enhance the existing data in the FAQ database, and extract FAQs and split document paragraphs from the government service documents passed in by the government service API interface;

[0081] S200: for a user's query request query, use the StructBert algorithm to retrieve the FAQ question-answer pair database and return the target government affairs document with the highest similarity, use the StructBert algorithm to retrieve a number of FAQ question-answer pairs corresponding to the target government affairs document and return the target FAQ question-answer pairs with a similarity higher than a preset threshold;

[0082] S300: When there is no target FAQ question-answer pair with a similarity higher than a preset threshold in the target government affairs document, based on the FAQ question-answer pair and document paragraph corresponding to each government affairs document, the query request query and each government affairs document are subjected to a three-branch decision hybrid retrieval including bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval and dynamic self-attention DynAttenBiLSTM content retrieval to obtain a hybrid retrieval result;

[0083] S400: The mixed search result is transmitted to the government affairs answering big model trained with the government affairs service document, and the question and answer result of the query request query is outputted.

[0084] It should be noted that the current intelligent question and answer system of the government service network mainly relies on the response support of existing question and answer data in the traditional sense and manual replies to user questions. Government service documents and policy documents are not fully utilized and mined. Manual replies are time-consuming and costly. In addition, the query performance of traditional retrieval methods is significantly reduced when dealing with massive data. The generalization performance is highly dependent on database retrieval information, and the recalled answers are difficult to meet users' search needs for precision and high relevance.

[0085] In order to solve the above problems, this embodiment uses a large model to enhance the existing data in the FAQ database, extracts FAQ pairs and splits document paragraphs for government service documents transmitted by the government service network API interface, uses the StructBert algorithm to retrieve FAQ pairs of the query request query, and outputs them when the similarity is higher than a preset threshold. Otherwise, the query request query and each government service document are subjected to a three-branch decision hybrid search, including bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval and dynamic self-attention DynAttenBiLSTM content retrieval, to obtain a hybrid retrieval result. Finally, the government service answer large model is used to output the question and answer result of the query request query based on the hybrid retrieval result. Thus, by integrating the three-branch decision modules and the hybrid retrieval module of multiple similarity algorithms, the document answer can be traced, the answer reliability is strong, and the accuracy is high.

[0086] In order to explain the present invention more clearly, a specific example of a government affairs intelligent question-answering system using the government affairs intelligent question-answering method is provided below. The government affairs intelligent question-answering system includes: a data storage unit, a FAQ question-answering retrieval unit, a RAG+LLM hybrid retrieval unit, and a reply unit. Among them:

[0087] The data storage unit is used to clean and preprocess the source data of government service documents and store them in a database, including FAQ question and answer database preprocessing, classification processing of service document source files by region and subject, cleaning of doc source files and service document attachment materials and storing them in a database unit.

[0088] The FAQ question and answer retrieval unit is used to preferentially retrieve the existing question and answer pairs of the user question query in the database, and preferentially return the answers to the question and answer pairs existing in the database, wherein the first retrieval module is a two-stage hybrid retriever and the retrieval algorithm is the government affairs Structbert algorithm.

[0089] The RAG+LLM hybrid retrieval unit is used to retrieve government service document knowledge base data to make up for the absence of question and answer pairs with high correlation thresholds in the FAQ question and answer database. Among them, the second retrieval module is a multi-layer algorithm module fusion retrieval based on three-branch decision-making, including bce-embedding-base-V1 vectorized retrieval and refined ranking, BM25-scSentenceTransformer hybrid retrieval, dynamic self-attention DynAttenBiLSTM content retrieval, and three-branch decision-making module.

[0090] The reply unit is used to input the results obtained from the hybrid retrieval into the large model answer module trained on the government affairs documents to output an answer for replying the user's query.

[0091] In practical applications, in the data storage unit module, the specific processing steps are as follows:

[0092] Step 1: Perform data enhancement on the FAQ database. Since there is currently little historical government affairs Q&A data, we first use the big model to rewrite and expand the historical queries in the database one by one, expanding a single question into three semantically similar queries, thereby enhancing the generalization performance of the government affairs Q&A system on the FAQ database.

[0093] Step 2: Extract and display fields from the data passed in by the government service API interface according to a fixed template format. Considering that government service documents are lengthy and contain a lot of information, the data passed in by the government service API interface is extracted, cleaned, and stored in the database. When the system matches the government service document that the user wants to retrieve, it will give priority to returning the summarized key fixed information, which includes four categories: acceptance conditions, handling procedures, required materials, and basic information. The extracted information is stored in the FAQ database in the form of question-service document name, answer-four types of key fixed information content.

[0094] Step 3: Preprocess government service documents. Load the government service doc type source document, use the python-docx library to read and clean the document. Since there are a lot of table data in government service documents, the python-docx library can effectively identify document titles and table information. Therefore, use this module to read and process docx files, split them into single document paragraphs and indexes according to the secondary titles, remove stop words, segment the text, vectorize it, and store it in the vector database.

[0095] In practical applications, the implementation of the FAQ question and answer retrieval unit includes the following specific steps:

[0096] Step 1: First-stage government affairs document title retrieval, based on FAQ database using Structbert algorithm retrieval. According to the FAQ question-answer database, a government affairs domain-specific Structbert model is trained, and the Structbert algorithm is used to search the FAQ database for user queries first and return the similarity above the threshold. The title of the service document.

[0097] Step 2: Second-stage government service document FAQ content retrieval. Use the Structbert algorithm module to retrieve the FAQ question-answer data corresponding to the service document classification obtained in step 1, and return the database question-answer answers that meet the threshold conditions.

[0098] For the scStructBert retrieval decision module, the similarity between the user question and the FAQ database is calculated based on the scStructBert model. The specific processing steps are as follows:

[0099] S1: Use the Qwen big model to generate 5 paragraph-level question-answer pairs for each document paragraph of the government service document. At the same time, use the Qwen big model to generate 5 question-answer pairs with summary content and answers across paragraphs based on the entire document. At the same time, the FAQ database question-answer pairs are loaded as training data for preprocessing.

[0100] S2: Prepare the data into a standard text-label input format and split the data into training set: validation set: test set = 8:1:1 ratio;

[0101] S3: The StructBert model is an extension of the pre-trained BERT language model. Based on the BERT model, a multi-layer bidirectional transformer network is used to reorganize a certain number of tokens after word masking and predicting the correct order to amplify the masked LM task capability. At the same time, the next sentence and the previous sentence are predicted as new sentence prediction tasks, so that the model can explicitly capture the fine-grained word structure in a single sentence while also correctly modeling the structure between sentences in a bidirectional manner. In this step, the training set and validation set data processed by S2 are used as the input of the StructBert model, random_seed=8 is set, the model is iterated until the loss value tends to be stable, the trained model weight file is saved to prepare the model for the downstream prediction task, and finally the optimal model weight is loaded, and the output after the test set data is input is used as the evaluation criterion:

[0102] ;

[0103] Among them, cosine similarity is the evaluation criterion, A and B are the vectorized sentence pairs, and n represents the vectorization dimension.

[0104] S4: Initial search based on semantic similarity of government service documents: Load the model weights trained in S3, predict the semantic similarity between the user input query and the names of all government service documents in the FAQ database, perform the initial relevance search of the database, and obtain the most relevant government service documents;

[0105] S5: Secondary search based on semantic similarity of the question and answer corpus corresponding to a specific document: Search the name of the government service document obtained in S4 corresponding to the FAQ question and answer database, use the model in step S3 to calculate the semantic similarity between the user input question and the query in the question database corresponding to the initial search of the government service document, calculate the score, and return the score greater than the threshold result.

[0106] In practical applications, the implementation of the RAG+LLM hybrid retrieval unit specifically includes bce-embedding-base-V1 vectorized retrieval and refined ranking, BM25-scSentenceTransformer hybrid retrieval, dynamic self-attention DynAttenBiLSTM content retrieval, and three decision modules. Among them:

[0107] The specific steps of the bce-embedding-base-V1 vectorized retrieval and refined ranking module are as follows: (1) The document is divided into individual document paragraphs according to the secondary title; (2) The bce-embedding-base-V1 vectorized knowledge base text documents and queries are used; (3) The cosine similarity is used to calculate the semantic similarity between the query and the vector knowledge base to obtain the top 10 content module retrieval results with the highest similarity; (4) The bce-reranker-base_v1 model is used to rank the 10 most relevant paragraph documents retrieved and assign weights according to the ranking to obtain the score vector .

[0108] The specific steps of the BM25-scSentenceTransformer hybrid retrieval module are as follows: (1) First-stage title retrieval and positioning of government service documents: Use the improved BM25 algorithm to retrieve the correlation between the user query and the title of the government service document based on probability, and return the 7 most relevant service items; (2) Train the scSentenceTransformer semantic model in the government service field for all government service documents; (3) Second-stage document content retrieval: Use the scSentenceTransformer semantic model trained in (2) to predict the correlation between the query and the paragraph-level documents of the top 10 government service documents, and return the most relevant top 10 paragraph text content and corresponding score vectors. .

[0109] Use cosine similarity to calculate the semantic similarity between the query and the vector knowledge base. The specific operation process is as follows:

[0110] a. For government affairs documents, construct paragraph sentence alignment samples based on paragraph-level titles, vectorize similar sentences based on the pre-trained model "distiluse-base-multilingual-cased", and calculate similarity; ;

[0111] in, are the vectorized sentence pairs, represents the vectorized dimension, is the correlation weight, for positive samples Take a higher threshold.

[0112] b. In order to increase the diversity of the dataset and balance the labels, the original high-similar sentence pairs are randomly shuffled to form sentence pair negative samples for data enhancement. The paragraph-level document content with low relevance is randomly selected to form sentence pairs, and the sentence pairs are vectorized, and similarity labels with lower thresholds are set.

[0113] The implementation path of the improved BM25 algorithm is as follows:

[0114] S1: Calculate the improved inverse document frequency IDF. Use the weighted global word frequency statistical method to calculate the IDF value based on the word frequency information of multiple topics in government service documents, thereby improving the cross-domain retrieval capabilities on multiple government service topics such as household registration, business license, housing security, and certificate processing. The calculation formula is as follows: ;

[0115] in, Indicates The total number of documents for a topic, is the total number of topics; Represents the weight of the i-th topic, weight coefficient , For the theme exist The total number of hits during the period, It is a government service The total number of clicks on the platform during the period, and meets The interface module of the government affairs platform regularly updates the number of clicks on each topic in the previous period as needed, thereby ensuring that the real-time cross-topic search results of the improved IDF meet the actual needs of users in that period.

[0116] S2: Calculate the improved term frequency TF. Based on the tabular structure characteristics of government service documents, the user query keywords after word segmentation only appear in the secondary title of the service document, with a low frequency. Therefore, in order to enhance the query's attention to the secondary title with a low word frequency, the present invention assigns different weights to different parts of the tabular service document to improve BM25's attention to the secondary title. The calculation formula is as follows.

[0117] ;

[0118] Among them, the weight given to the title , secondary title weight , the weight of the paragraph text .

[0119] S3: Calculate optimal hyperparameters and . Use decision tree DT combined with grid search to define the best hyperparameters for BM25 and , the technical route is:

[0120] 1. Initialize the decision tree DT model, ;

[0121] 2. For each , calculate the improved BM25 score , construct feature vector ,Will Add to the training set and use the training set to train the DT model;

[0122] 3. For each , , use the DT model to predict its relevance score on the validation set , calculation error , record the current optimal and , use the grid search method to globally search for the parameter combination corresponding to the minimum error ( *, *).

[0123] S4. The improved BM25 formula is calculated by combining structured paragraph information as follows.

[0124] ;

[0125] in, Indicates the number of terms in the query, Is the first terms, is the inverse document frequency, N is the total number of documents, Indicates that it contains a term The number of documents, Representing terms In the documentation The number of occurrences in represents a collection of terms, Representing terms In the documentation The frequency of terms in Representation Document Length, represents the average length of all documents, and is the adjustment parameter.

[0126] The dynamic self-attention DynAttenBiLSTM content retrieval module, the bidirectional long short-term memory network BiLSTM is an extension of the recurrent neural network RNN, which adds reverse operations on the basis of RNN, and uses memory units and gating mechanisms to memorize and use context information so that the model can predict the next word. BiLSTM combines bidirectional processing, long sequence dependency modeling and the use of context information, so that the algorithm can effectively predict the next word given a long sentence. Although the traditional self-attention mechanism can capture long dependencies in the sequence, it may ignore the contextual differences between the query and the paragraph. Therefore, in order to improve the retrieval accuracy, this module has designed a new dynamic self-attention mechanism, which automatically adjusts the weight according to the specific content of the query and government service documents, and accurately focuses on relevant information.

[0127] The specific processing steps are:

[0128] (1) In order to make the neural network pay more attention to key features, the DynAttenBiLSTM model is built in combination with the dynamic self-attention layer mechanism. The dynamic query encoding is used in the self-attention layer to dynamically calculate the similarity between the query and the document content, and the position encoding is used to dynamically focus on the key paragraphs during the pre-selected similarity calculation, thereby adjusting the attention weight. This module embeds the attention mechanism into the BiLSTM output layer, thereby weighting the input through the attention score to focus on the most relevant information of the query. The attention score calculation formula is as follows:

[0129] ;

[0130] Q, K, and V are matrices of query, key, and value, respectively. is the dimension of the key.

[0131] (2) When calculating cosine similarity, in addition to considering text features, this module introduces position encoding between queries and paragraphs to help the model understand the impact of the position of certain keywords in the query on the importance of the paragraph. The specific approach is: assuming that the query is encoded as q, the paragraph is encoded as p, and the relative position encodings between the query and the paragraph are PEq and PEp, first use the formula to add the position encoding to the original word vector to obtain the enhanced word vector, and then use the enhanced cosine similarity formula to calculate the similarity between the query and the government affairs paragraph-level document.

[0132] ;

[0133] ;

[0134] (3) Based on the result of step (2), the top 10 most relevant paragraph text contents and corresponding score vectors are returned. .

[0135] The improved three-branch decision module is mainly used to convert the above-mentioned bce-embedding-base-V1 vectorized retrieval and refined ranking, BM25-scSentenceTransformer hybrid retrieval, and dynamic self-attention DynAttenBiLSTM content retrieval corresponding results Make the final decision and recall the final paragraph-level document retrieval results. , and dynamically adjust the weight influence of the above three methods by analyzing the user's historical query information while recalling the search content. The specific operation process is:

[0136] (1) Standardization of preliminary ranking results. For the preliminary candidate results U of the three retrieval methods, bce-embedding-base-V1 vectorized retrieval and refined ranking, BM25-scSentenceTransformer hybrid retrieval, and dynamic self-attention DynAttenBiLSTM, for each vector A normalization formula is used to map the scores of different scoring criteria of different retrieval methods to the same scale.

[0137] ;

[0138] (2) Intent matching weighting and dynamic correction. Define the weight vector Focus on adjusting the influence ratio of each method on the results, take the user's historical government consultation information into consideration, and use the big model to classify the historical query intent, including: entry and exit, marriage registration, household registration, transportation, etc., to form a sequence , dynamically adjust the weight formula based on the user’s historical government affairs consultation messages.

[0139] ;

[0140] ;

[0141] in, represents the initial weight value, is a correction coefficient used to balance the influence of historical government dialogue information on current decision-making; It is the user's historical conversation information and current The calculated weight correction function is processed by the intention matching weighted method; Indicates the click weight of the paragraph type. For example, if users click on the BM25-scSentenceTransformer algorithm result more often, the weight of the corresponding paragraph will be increased. and Represents the user's historical government dialogue and the current one judged by the big model Intent category.

[0142] (3) Calculate the fusion score. According to the dynamically adjusted weights Calculate the fusion score for the candidate set U, where , They represent the bce-embedding-base-V1 vectorized retrieval and refined ranking, BM25-scSentenceTransformer hybrid retrieval, and DynAttenBiLSTM models respectively.

[0143] ;

[0144] (4) According to the fusion score, the candidate set U is added to the three decision classifiers.

[0145] ;

[0146] Among them, H represents a high confidence set, L represents a low confidence set, and M represents a fuzzy set. represents the high confidence fusion score threshold, Represents the low confidence fusion score threshold.

[0147] (5) Fuzzy set secondary decision. The fuzzy paragraph set M is sorted again using the user's historical information, rearranged using the formula, and the first fuzzy paragraph after the rearrangement is selected to be added to the high confidence result set H, and the final high confidence paragraph result is output. .in is the weighted coefficient of semantic similarity, Indicates the current user's inquiry question Semantic vector of historical government affairs dialogue with users The cosine similarity of .

[0148] ;

[0149] In practical applications, the reply unit module is used to convert the high confidence paragraph results predicted by the three decision modules into Push it to the big model trained with government service document data, call the big model prompt project, guide the trained big model to generate answers to questions, and finally generate text and output answers.

[0150] Reference Figure 3 , Figure 3 This is a structural block diagram of an embodiment of the government affairs intelligent question-and-answer device of the present invention.

[0151] like Figure 3 As shown, the government affairs intelligent question-answering device proposed in the embodiment of the present invention includes:

[0152] Processing module 10, used to enhance the existing data in the FAQ database using the big model, and extract FAQs and split document paragraphs from the government service documents transmitted by the government service network API interface;

[0153] The first retrieval module 20 is used to retrieve the FAQ question-answer pair database using the StructBert algorithm for the user's query request query and return the target government affairs document with the highest similarity, and retrieve several FAQ question-answer pairs corresponding to the target government affairs document using the StructBert algorithm and return the target FAQ question-answer pairs with a similarity higher than a preset threshold;

[0154] The second retrieval module 30 is used for executing the three-branch hybrid retrieval of the query request query and each government service document including bce-embedding-base-V1 vectorization retrieval, BM25-scSentenceTransformer hybrid retrieval and dynamic self-attention DynAttenBiLSTM content retrieval based on the FAQ question-answer pair and document paragraph corresponding to each government service document when there is no target FAQ question-answer pair with a similarity higher than a preset threshold value in the target government service document to obtain a hybrid retrieval result;

[0155] The output module 40 is used to transmit the mixed search results to the government affairs answering model trained with government affairs documents, and output the question and answer results of the query request query.

[0156] Other embodiments or specific implementation methods of the government affairs intelligent question and answer device of the present invention can refer to the above-mentioned method embodiments and will not be repeated here.

[0157] In addition, the present invention also proposes a government affairs intelligent question and answer device, which includes: a memory, a processor, and a government affairs intelligent question and answer program stored in the memory and executable on the processor, and when the government affairs intelligent question and answer program is executed by the processor, the steps of the government affairs intelligent question and answer method described above are implemented.

[0158] The specific implementation methods of the government affairs intelligent question and answer device of the present application are basically the same as the embodiments of the above-mentioned government affairs intelligent question and answer method, and will not be repeated here.

[0159] In addition, the present invention also proposes a readable storage medium, which includes a computer-readable storage medium on which a government affairs intelligent question-answering program is stored. The readable storage medium may be Figure 1The memory 1005 in the terminal may also be at least one of ROM (Read-Only Memory) / RAM (Random Access Memory), a magnetic disk, and an optical disk. The readable storage medium includes a number of instructions for enabling a government affairs intelligent question and answer device with a processor to execute the government affairs intelligent question and answer method described in each embodiment of the present invention.

[0160] The specific implementation methods in the readable storage medium of the present application are basically the same as the embodiments of the above-mentioned government affairs intelligent question and answer method, and will not be repeated here.

[0161] It is understood that, in the description of this specification, the description with reference to the terms "one embodiment", "another embodiment", "other embodiments", or "first to Nth embodiments" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0162] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0163] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0165] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A government affairs intelligent question-answering method, characterized in that: The following steps are involved: Use the big model to enhance the existing data in the FAQ database, and extract FAQs and split document paragraphs from government service documents passed in through the government service API interface; For the user's query request query, the SC-StructBert algorithm is used to retrieve the FAQ question-answer pair database and return the target government service document with the highest similarity, and the SC-StructBert algorithm is used to retrieve several FAQ question-answer pairs corresponding to the target government service document and return the target FAQ question-answer pairs with a similarity higher than a preset threshold; When there is no target FAQ question-answer pair with a similarity higher than a preset threshold in the target government affairs document, based on the FAQ question-answer pair and document paragraph corresponding to each government affairs document, the query request query and each government affairs document are subjected to a three-branch decision hybrid search including bce-embedding-base-V1 vectorized search, BM25-scSentenceTransformer hybrid search and dynamic self-attention DynAttenBiLSTM content search to obtain a hybrid search result; specifically, the following steps are performed: Execute the query request query and the bce-embedding-base-V1 vectorized retrieval of each government service document: Calculate the semantic similarity between the query request query and several word vectors corresponding to each document paragraph in the vector database, use bce-embedding-base-V1 to sort the document paragraphs that meet the semantic similarity requirements, and obtain the score vector ; Perform BM25-scSentenceTransformer hybrid retrieval of the query request query and each government service document: Use the improved BM25 algorithm to retrieve the correlation between the query request query and the document name of the government service document based on probability, return the government service documents that meet the first correlation requirement, and use the government service documents that meet the correlation requirement to train the scSentenceTransformer semantic model, use the trained scSentenceTransformer semantic model to predict the correlation between the query request query and the document paragraphs of the government service document that meet the second correlation requirement, and return the document paragraphs that meet the third correlation requirement and the corresponding score vector ; Execute dynamic self-attention DynAttenBiLSTM content retrieval between query request query and each government service document: Build the DynAttenBiLSTM model in combination with the dynamic self-attention layer mechanism, use dynamic query encoding in the self-attention layer to dynamically calculate the similarity between the query and the document content, and dynamically focus on key paragraphs through position encoding when pre-selecting similarity calculations to adjust the attention weight; then introduce the position encoding between the query request query and the document paragraphs of the government service document, add the position encoding to the original word vector to obtain the enhanced word vector, and then use the enhanced cosine similarity to calculate the similarity between the query request query and the document paragraphs of the government service document, and return the document paragraphs that meet the fourth relevance requirements and the corresponding score vector. ; Perform a three-branch decision hybrid search including bce-embedding-base-V1 vectorized search, BM25-scSentenceTransformer hybrid search, and dynamic self-attention DynAttenBiLSTM content search of the score vector to obtain a hybrid search result; The hybrid search results are transmitted to the government affairs answering big model after fine-tuning by the government affairs service documents, and the question-answering results of the query request query are obtained by output.

2. The government affairs intelligent question-answering method according to claim 1, characterized in that: The big model is used to enhance the data in the FAQ database, and the FAQ extraction and document segmentation steps are performed on the government service documents passed in by the government service API interface, including: For the existing FAQ pairs in the FAQ database, the big model is used to expand the questions in each FAQ pair into several semantically similar query requests, and the questions of each existing FAQ pair are replaced with the corresponding generated query requests with similar semantics and stored in the FAQ database, thus realizing data enhancement of the FAQ database. Extract key fixed information of the government service document consisting of acceptance conditions, handling procedures, required materials and basic information from the government service document transmitted through the government service network API interface according to a fixed template format, use the document name of the government service document as a question, use the key fixed information of the document as an answer, and construct a FAQ question-answer pair for the government service document; Load the doc type source document of the government service document, use the python-docx library to read and clean the document, split it into individual document paragraphs and indexes according to the secondary titles, remove stop words, segment and vectorize the text, and store it in the vector database.

3. The government affairs intelligent question-answering method according to claim 2, characterized in that: For a user's query request query, the FAQ question-answer pair database is retrieved using the SC-StructBert algorithm and the target government affairs service document with the highest similarity is returned. The steps of retrieving several FAQ question-answer pairs corresponding to the target government affairs service document using the SC-StructBert algorithm and returning the target FAQ question-answer pairs with a similarity higher than a preset threshold are specifically included: The initial SC-StructBert model is trained according to the FAQ question and answer database to obtain the SC-StructBert algorithm model for government affairs intelligent question and answer; Using the sc-structbert algorithm model, a semantic similarity search is performed between the user's query request query and the document name of the government affairs document corresponding to each FAQ question and answer database, and the target government affairs document with the highest similarity is returned; The sc-structbert algorithm model is used to perform semantic similarity retrieval on a plurality of FAQs in a database of the user's query request query and the FAQs corresponding to the target government affairs document, and a target FAQ pair with a similarity higher than a preset threshold is returned.

4. The government affairs intelligent question-answering method according to claim 3, characterized in that: The initial SC-StructBert model is trained according to the FAQ question-answer database to obtain the SC-StructBert algorithm model steps for government affairs intelligent question-answering, specifically including: For each government service document corresponding to the FAQ question-answer pair database, after the document paragraphs are split, the big model is used to generate several paragraph-level question-answer pairs for each document paragraph. At the same time, the big model is used to generate several cross-paragraph question-answer pairs for the entire government service document. Prepare a number of paragraph-level question-answer pairs, a number of cross-paragraph question-answer pairs, and a number of FAQ question-answer pairs stored in each FAQ question-answer pair database as sample data, and divide the sample data into a training set, a validation set, and a test set according to an 8:1:1 ratio; The training set and validation set are used as the input of the initial SC-StructBert model for training until the model converges, and the test set is used to test and evaluate the trained SC-StructBert algorithm model.

5. The government affairs intelligent question-answering method according to claim 1, characterized in that: The steps of using the improved BM25 algorithm to retrieve the correlation between the query request query and the document name of the government affairs document based on probability include: Using the weighted global word frequency statistics method, the inverse document frequency (IDF) is improved based on the word frequency information of several topics in government affairs documents. Based on the table types in government affairs documents, different weights are assigned to different parts of the table-type government affairs documents, and the improved term frequency TF is calculated; Use decision tree DT combined with grid search to define the best hyperparameters for BM25 and , and combined with structured paragraph information to improve the BM25 calculation formula as follows: ； in, Indicates the number of terms in the query, Is the first terms, is the inverse document frequency, N is the total number of documents, Indicates that it contains a term The number of documents, Representing terms In the documentation The number of occurrences in Representation Document Length, represents the average length of all documents, and is the adjustment parameter.

6. The government affairs intelligent question-answering method according to claim 1, characterized in that: Execute the three-branch decision hybrid retrieval steps including bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval and dynamic self-attention DynAttenBiLSTM content retrieval of the score vector, including: Score vectors for bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval, and dynamic self-attention DynAttenBiLSTM content retrieval The normalization formula (f) is used to map the scores of different scoring criteria of different retrieval methods to the same scale; ； Define the weight vector Adjust the impact of each method on the results; ； ； In the formula, represents the initial weight value, is the correction factor, It is the user's historical conversation information and current The calculated weight correction function, Indicates the click weight of the paragraph type. and Represents the user's historical government dialogue and the current one judged by the big model Intent category; According to the dynamically adjusted weight Calculate the fusion score for the candidate set U; ； In the formula, , They represent three models: bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval, and dynamic self-attention DynAttenBiLSTM content retrieval. According to the fusion score, the candidate set U is added to the three decision classifiers; ； In the formula, H represents the high confidence set, L represents the low confidence set, and M represents the fuzzy set; The fuzzy paragraph set M is sorted and rearranged using user history information, and the first fuzzy paragraph after rearrangement is selected to be added to the high confidence result set H, and the final high confidence paragraph result is output. ; The rearranged expression is as follows: ； is the weighted coefficient of semantic similarity, Indicates the current user's inquiry question Semantic vector of historical government affairs dialogue with users The cosine similarity of .

7. An intelligent government question-answering device, characterized in that: The government affairs intelligent question-answering method according to any one of claims 1 to 6 comprises: The processing module is used to enhance the existing data in the FAQ database using the big model, and to extract FAQs and split document paragraphs from the government service documents passed in by the government service API interface; The first retrieval module is used to retrieve the FAQ question-answer pair database using the SC-StructBert algorithm for the user's query request query and return the target government affairs service document with the highest similarity, and to retrieve a number of FAQ question-answer pairs corresponding to the target government affairs service document using the SC-StructBert algorithm and return the target FAQ question-answer pairs with a similarity higher than a preset threshold; The second retrieval module is used for executing the three-branch decision hybrid retrieval of the query request query and each government service document including bce-embedding-base-V1 vectorized retrieval, BM25-scSentenceTransformer hybrid retrieval and dynamic self-attention DynAttenBiLSTM content retrieval based on the FAQ question-answer pair and document paragraph corresponding to each government service document when there is no target FAQ question-answer pair with a similarity higher than a preset threshold in the target government service document to obtain a hybrid retrieval result; The output module is used to transmit the mixed search results to the government affairs answering model that has been fine-tuned by the government affairs service documents, and output the question and answer results of the query request query.

8. An intelligent government question-answering device, characterized in that: The government affairs intelligent question and answer device includes: a memory, a processor, and a government affairs intelligent question and answer program stored in the memory and executable on the processor. When the government affairs intelligent question and answer program is executed by the processor, the steps of the government affairs intelligent question and answer method as described in any one of claims 1 to 6 are implemented.

9. A storage medium, characterized in that: The storage medium stores an intelligent government question-and-answer program, which, when executed by a processor, implements the steps of the intelligent government question-and-answer method as described in any one of claims 1 to 6.

Images