An intelligent government affairs question-answering system based on big data
By introducing dependency characteristics and multiple attention mechanisms in the government Q&A system, the difficulty of deep semantic understanding of existing systems when dealing with complex text statement structures is solved, and the accuracy of answer positioning and user experience are improved.
Patent Information
- Application Number
- CN202310362407.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-07
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-04-07
AI Technical Summary
The existing government affairs question and answer system is difficult to understand deep semantics when dealing with complex text statement structures, and the pre-training model is trained for a long time and lacks text syntax spatial information, resulting in inaccurate answer positioning.
An intelligent government affairs question and answer system based on big data was designed, including text analysis module, feature extraction module, information interaction module and answer prediction module. Through dependency characteristics, the system can learn the central intention and article topics, and extract it in combination with multiple attention mechanisms. Text vector representation to improve the accuracy of answer positioning.
By introducing dependency characteristics, the system can more accurately understand the deep semantics of the text, improve the accuracy of answer positioning of the Q&A system, and improve the user experience.
Smart Images

Figure CN118410132B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of machine reading comprehension of natural language processing, and specifically relates to an intelligent government affairs question and answer system based on big data. Background Art
[0002] Machine reading comprehension means that given a question, the machine can provide the correct answer.
[0003] With the rapid development of information technology, human society has entered the information age, and higher requirements have been placed on the comprehensive information service capabilities of existing government systems. When faced with query requests from users, they all hope that the model can quickly return an answer that meets the user's question. For users, it can improve their experience and satisfaction with the government system, and for the government system, it can shape a good government image and reputation.
[0004] At present, most of the work is focused on how to improve the pre-trained model to enhance the level of the question-answering system. However, the training time of the pre-trained model is relatively long, and the pre-trained model lacks consideration of the text syntactic space information. Therefore, how to build a more accurate and intelligent government question-answering system has broad application prospects. Summary of the invention
[0005] In order to solve the above technical problems, the present invention proposes an intelligent government affairs question-answering system based on big data, the system comprising: a text parsing module, a feature extraction module, an information interaction module, and an answer prediction module;
[0006] The text parsing module is used to calculate the correlation between the user's question Q and the documents in the system text library, sort them according to the correlation, and select the top K documents with the highest correlation as the recalled documents P;
[0007] The feature extraction module is used to extract the dependency feature H between the recalled document P and the user question Q syn ;
[0008] The information interaction module is used to recall the document P, the user question Q and the dependency feature H syn Perform multiple attention information interactions to obtain the final text vector representation H oyt ;
[0009] The answer prediction module is used to represent the final text vector H out The fully connected layer is used to predict the starting and ending positions of the final answer.
[0010] Beneficial effects of the present invention:
[0011] The present invention introduces dependency relationships to help the system learn the central intention of the question and the main idea of the article, solving the problem that the complex sentence structure relationships in the article make it difficult for the system to understand the deep semantics of the text;
[0012] The present invention uses multiple attention mechanisms to enable the system to pay more attention to the content related to the question and document and ignore irrelevant content; it incorporates dependency features to deeply explore the connection between the question and the answer in the document, thereby improving the positioning of the standard answer in the question-answering system and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 This is a schematic diagram of an intelligent government affairs question and answer system based on big data according to the present invention. DETAILED DESCRIPTION
[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0015] An intelligent government question-answering system based on big data, such as Figure 1 As shown, the system includes: a text parsing module, a feature extraction module, an information interaction module, and an answer prediction module;
[0016] The text parsing module is used to calculate the correlation between the user's question Q and the documents in the system text library, sort them according to the correlation, and select the top K documents with the highest correlation as the recalled documents P;
[0017] The feature extraction module is used to extract the dependency feature H between the recalled document P and the user question Q syn ;
[0018] The information interaction module is used to recall the document P, the user question Q and the dependency feature H syn Perform multiple attention information interactions to obtain the final text vector representation H out ;
[0019] The answer prediction module is used to represent the final text vector H out The fully connected layer is used to predict the starting and ending positions of the final answer.
[0020] The formula for calculating the relevance of question Q to each document is as follows:
[0021]
[0022]
[0023]
[0024] Among them, similarity(Q,P) represents the correlation between question Q and document P, and W i Indicates q i The weight of document P, q i represents each character in question Q, N represents the number of all documents, n(q i ) means including q i The number of documents, score(q i ,P) represents q i The relevance to document P, lenq represents the length of question Q, len(P) represents the length of document P, avgp represents the average length of the document, k represents the hyperparameter used to control the range of word frequency in the article, and b represents the hyperparameter used to control the range of information content in the article length.
[0025] The dependency feature of the text H syn The extraction steps are as follows:
[0026] The recalled document P is concatenated with the user question Q, and then input into the BERT model after the original encoding method is improved for word vector encoding to obtain the word vector representation H of the text. last ;
[0027] The dependency relationship of the concatenated text is obtained according to the syntactic dependency tree, and the adjacency matrix M is constructed according to the text dependency relationship, that is, whether there is an edge between words. ij , and then mapped into vector representation according to the dependency information between words;
[0028] According to the adjacency matrix M ij Calculate the attention distribution between words;
[0029] According to the adjacency matrix M ij The vector obtained by mapping the dependency information between words represents the attention distribution of the dependency information between words;
[0030] According to the above two attention distributions, the word vector representation H last Do weighted summation to get the text dependency feature H syn .
[0031] Improve the original encoding method of the BERT model, including:
[0032]
[0033] in, Indicates the improved encoding method, p irepresents the absolute encoding vector at position i, p1 is the absolute encoding vector of the first position, and α represents a constant of α∈[0,1]andα≠0.5.
[0034] Furthermore, the attention distribution between words is calculated, including:
[0035]
[0036]
[0037]
[0038] Among them, att ij Represents the attention distribution between words, represents the attention score of the relationship between node i and node j, W a1 With W a2 They represent the first and second trainable weight coefficients for calculating the attention distribution between words, LeakyReLU() represents the activation function, and h i represents the word vector representation of node i, W represents the third trainable weight coefficient for calculating the attention distribution of the relationship between words, T represents the transpose of the matrix, Represents the relationship attention score between node i and node j after integrating the adjacency matrix information, Indicates that M is provided ij Adjacency matrix information, when M ij The original value is retained for the position of 1. ij The position of 0 is assigned a value of 1e-9, M i · represents the node subscript set that has a relationship with node i, and exp() represents the exponential function with the natural constant e as the base.
[0039] Calculate the attention distribution of dependency information between words, including:
[0040]
[0041]
[0042]
[0043] in, Represents the attention distribution of the dependency information between words, represents the dependency information attention score of node i and node j, W1 and W2 represent the first and second trainable weight coefficients for calculating the dependency information attention distribution between words, b1 and b2 represent the first and second bias terms, r ij A vector representation representing the dependency information mapping, represents the attention score of the dependency information between node i and node j after integrating the adjacency matrix information, Indicates that M is provided ij Adjacency matrix information, when M ij The original value is retained for the position of 1. ij The position of 0 is assigned a value of 1e-9, exp() represents an exponential function with the natural constant e as the base, N i Indicates the length of the text.
[0044] According to the above two attention distributions, the word vector representation H last Do a weighted sum, including:
[0045]
[0046]
[0047]
[0048] H last =[h1,h2,…,h j …,h n ]
[0049] in, represents the dependency feature representation of node i, represents the relationship information representation of node i, represents the dependency information representation of node i, M i · represents the node subscript set that has a relationship with node i, W att With W rel Respectively represent the first and second trainable weight coefficients for weighted summation, att ij Represents the attention distribution between words, represents the attention distribution of the dependency information between words, h j Represents the word vector representation of node j.
[0050] The document P, user question Q and dependency feature H will be recalled syn Perform multiple attention information interactions to obtain the final text vector representation H out ,include:
[0051] Calculate the attention distribution of question Q for document P, integrate the information of question Q into the representation of document P, and obtain the new document representation P ~ ;
[0052] Calculate the attention distribution of document P for question Q, integrate the information of document P into the representation of question Q, and obtain a new question representation Q ~ ;
[0053] Represent the document P ~ , Problem Representation Q ~ and H syn Perform splicing and use the self-attention mechanism to interact the dependency information with the text representation to obtain the text vector representation H out .
[0054] Calculate the new document representation P ~ ,include:
[0055] P ~ =H Q softmax(S P )
[0056] S P =ReLU(W′H Q ) T ReLU(W″H P )
[0057] Among them, S P represents the attention score of question Q for document P, H Q Indicates H laat The representation of question Q in H P Indicates H laat The representation of document P in , W′ and W″ respectively represent the first and second trainable weight coefficients of calculating the attention score of question Q for document P, ReLU() represents the activation function, softmax() represents the normalization operation, and T represents the matrix transpose.
[0058] Calculate the new problem representation Q ~ ,include:
[0059] Q ~ =H P softmax(s Q )
[0060] S Q =W″′[H Q ;H P ;H Q °H P ]
[0061] Among them, S Q represents the attention score of document P for question P, H Q Indicates H last The representation of question Q in H P Indicates H last The representation of document P in , W″′ represents the trainable weight coefficient for calculating the attention score of document P for question P, softmax() represents the normalization operation, and ° represents the Hadamard product operation.
[0062] Document Representation ~ , Problem Representation Q ~ and H syn Perform splicing and use the self-attention mechanism to interact the dependency information with the text representation to obtain the text vector representation H out ,include:
[0063] H out =softmax((W Q H fs )((W K H fs ) T )(W V H fs )
[0064] H fs =[P ~ ;Q ~ ,H syn ]
[0065] Among them, H fs represents the concatenated text features, W Q , W K , W V Respectively represent the first, second, and third trainable weight coefficients for information interaction.
[0066] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. An intelligent government affairs question-answering system based on big data, characterized in that: include: Text parsing module, feature extraction module, information interaction module, answer prediction module; The text parsing module is used to calculate the correlation between the user's question Q and the documents in the system text library, sort them according to the correlation, and select the top K documents with the highest correlation as the recalled documents P; The feature extraction module is used to extract the dependency feature H between the recalled document P and the user question Q syn ; Extract the dependency feature H between the recalled document P and the user question Q syn ,include: The recalled document P is concatenated with the user question Q, and then input into the BERT model after the original encoding method is improved for word vector encoding to obtain the word vector representation H of the text. last ; According to the syntactic dependency tree, the dependency relationship of the concatenated text is obtained, and the adjacency matrix M is constructed according to the text dependency relationship ij , and then mapped into vector representation according to the dependency information between words; According to the adjacency matrix M ij Calculate the attention distribution between words; According to the adjacency matrix M ij The vector obtained by mapping the dependency information between words represents the attention distribution of the dependency information between words; According to the above two attention distributions, the word vector representation H last Do weighted summation to get the text dependency feature H syn ; Calculate the attention distribution between words, including: Among them, att ij Represents the attention distribution between words, Represents the attention score of the relationship between node i and node j, W a1 With W a2 They represent the first and second trainable weight coefficients for calculating the attention distribution between words, LeakyReLU() represents the activation function, and h i represents the word vector representation of node i, W represents the third trainable weight coefficient for calculating the attention distribution of the relationship between words, T represents the transpose of the matrix, Represents the relationship attention score between node i and node j after integrating the adjacency matrix information, Indicates that M is provided ij Adjacency matrix information, when M ij The original value is retained for the position of 1. ij The position of 0 is assigned a value of 1e-9, M i. It represents the node subscript set that has a relationship with node i, and exp() represents the exponential function with the natural constant e as the base; The information interaction module is used to recall the document P, the user question Q and the dependency feature H syn Perform multiple attention information interactions to obtain the final text vector representation H out ; The answer prediction module is used to represent the final text vector H out The fully connected layer is used to predict the starting and ending positions of the final answer.
2. According to the big data-based intelligent government affairs question-answering system of claim 1, it is characterized in that: Calculate the relevance between the user's question Q and the documents in the system text library, including: Among them, similarity(Q,P) represents the correlation between question Q and document P, and W i Indicates q i The weight of document P, q i represents each character in question Q, N represents the number of all documents, n(q i ) means including q i The number of documents, score(q i ,P) represents q i Relevance to document P lenq represents the length of question Q, len(P) represents the length of document P, avgp represents the average length of the document, k represents the hyperparameter used to control the range of word frequency in the article, and b represents the hyperparameter used to control the range of information content in the article length.
3. According to the big data-based intelligent government affairs question-answering system of claim 1, it is characterized in that: Improve the original encoding method of the BERT model, including: in, Indicates the improved encoding method, p i represents the absolute encoding vector at position i, p1 is the absolute encoding vector of the first position, and α represents a constant of α∈[0,1]andα≠0.
5.
4. According to the big data-based intelligent government affairs question-answering system of claim 1, it is characterized in that: Calculate the attention distribution of dependency information between words, including: in, Represents the attention distribution of the dependency information between words, represents the dependency information attention score between node i and node j, W1 and W2 represent the first and second trainable weight coefficients for calculating the attention distribution of the dependency information between words, b1 and b2 represent the first and second bias terms, r ij A vector representation representing the dependency information mapping, represents the attention score of the dependency information between node i and node j after integrating the adjacency matrix information, Indicates that M is provided ij Adjacency matrix information, when M ij The original value is retained for the position of 1. ij The position of 0 is assigned a value of 1e-9, exp() represents an exponential function with the natural constant e as the base, N i Indicates the length of the text.
5. According to the big data-based intelligent government affairs question-answering system of claim 1, it is characterized in that: According to the above two attention distributions, the word vector representation H last Do a weighted sum, including: in, represents the dependency feature representation of node i, represents the relationship information representation of node i, represents the dependency information representation of node i, M i. Represents the node subscript set that has a relationship with node i, W att With W rel Respectively represent the first and second trainable weight coefficients for weighted summation, att ij Represents the attention distribution between words, represents the attention distribution of the dependency information between words, h j Represents the word vector representation of node j, H last =[h1,h2,…,h j …,h n ].
6. The intelligent government affairs question-and-answer system based on big data according to claim 1 is characterized in that: The document P, user question Q and dependency feature H will be recalled syn Perform multiple attention information interactions to obtain the final text vector representation H out ,include: Calculate the attention distribution of question Q for document P, integrate the information of question Q into the representation of document P, and obtain the new document representation P ~ ; Calculate the attention distribution of document P for question Q, integrate the information of document P into the representation of question Q, and obtain a new question representation Q ~ ; Represent the document P ~ , Problem Representation Q ~ and H syn Perform splicing and use the self-attention mechanism to interact the dependency information with the text representation to obtain the text vector representation H out .
7. The intelligent government affairs question-and-answer system based on big data according to claim 6 is characterized in that: Calculate the new document representation P ~ ,include: P ~ =H Q softmax(S P ) Among them, S P represents the attention score of question Q for document P, S P =ReLU(W′H Q ) T ReLU(W″H P ), H Q Indicates H last The representation of question Q in H P Indicates H last The representation of document P in , W′ and W″ respectively represent the first and second trainable weight coefficients of calculating the attention score of question Q for document P, ReLU() represents the activation function, softmax() represents the normalization operation, and T represents the matrix transpose.
8. The intelligent government affairs question-and-answer system based on big data according to claim 6 is characterized in that: Calculate the new problem representation Q ~ ,include: Q ~ =H P softmax(S Q ) Among them, S Q represents the attention score of document P for question P, S Q =W″′[H Q ;H P ;H Q °H P ], H Q Indicates H last The representation of question Q in H P Indicates H last The representation of document P in , W″′ represents the trainable weight coefficient for calculating the attention score of document P for question P, softmax() represents the normalization operation, and ° represents the Hadamard product operation.
