A question and answer method based on a gated global attention machine reading comprehension model

Abstract

The application belongs to the field of natural language processing, and particularly relates to a question and answer method based on a gated global attention machine reading comprehension model, which comprises the following steps: obtaining an article to be answered, performing feature extraction on the article, inputting article representation, question representation and option representation into an encoding layer respectively to obtain context information representation of the article, context information representation of the question and context information representation of the option; performing information interaction on the context information representation of the article, the context information representation of the question and the context information representation of the option by using an attention interaction layer to obtain deep interaction information of the question-article-answer; inputting the deep interaction information of the question-article-answer into an answer output layer to obtain a question result; the application helps the model to learn accurate information about the question by introducing global node information, and fully excavates logical information of the article and the question by introducing a three-attention mechanism, thereby improving the efficiency and accuracy of the intelligent question and answer system.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, specifically relating to a question-answering method based on a gated global attention machine reading comprehension model. Background Technology

[0002] Machine reading comprehension requires machines to read and understand human natural language text, and then answer questions related to the information in the text. This task is often used to measure a machine's natural language understanding ability. It can help humans quickly focus on relevant information from large amounts of text, reduce the cost of manual information retrieval, and has great application value in fields such as text question answering, information extraction, and dialogue systems.

[0003] Most current pre-trained models lack consideration for full-text information, simply inputting truncated sections of the article. Therefore, building a model that more accurately utilizes full-text and contextual information would greatly improve the efficiency and accuracy of question-answering systems. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention proposes a question-answering method based on a gated global attention machine reading comprehension model. The method includes: acquiring the article to be answered, inputting the article to be answered into the trained gated global attention machine reading comprehension model, and obtaining the question result; the gated global attention machine reading comprehension model includes an embedding layer, an encoding layer, an attention interaction layer, and an answer output layer.

[0005] The process of training a gated global attention machine reading comprehension model includes:

[0006] S1: Obtain the training dataset, which includes articles, questions, and options;

[0007] S2: Input the data in the training set into the embedding layer to extract features, and obtain the article representation, question representation and option representation;

[0008] S3: Input the article representation, question representation, and option representation into the encoding layer respectively to obtain the contextual information representation of the article, the contextual information representation of the question, and the contextual information representation of the option.

[0009] S4: An attention interaction layer is used to interact with the contextual information representations of the article, the question, and the options to obtain deep interactive information between the question, the article, and the answer.

[0010] S5: Input the deep interactive information of question-article-answer into the answer output layer to obtain the question result;

[0011] S6: Construct the model's loss function based on the problem results; adjust the model's parameters, and complete the model training when the loss function converges.

[0012] The beneficial effects of this invention are:

[0013] This invention improves the efficiency and accuracy of intelligent question answering systems by introducing global node information to help the model learn accurate information about the question and by introducing a three-attention mechanism to fully explore the logical information between the article and the question. Attached Figure Description

[0014] Figure 1 This is a flowchart of a question-answering method based on a gated global attention machine reading comprehension model according to the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] A question-answering method based on a gated global attention machine reading comprehension model, such as... Figure 1 As shown, the method includes: obtaining the article to be answered, inputting the article to be answered into the trained gated global attention machine reading comprehension model, and obtaining the question result; the gated global attention machine reading comprehension model includes an embedding layer, an encoding layer, an attention interaction layer, and an answer output layer.

[0017] The process of training a gated global attention machine reading comprehension model includes:

[0018] S1: Obtain the training dataset, which includes articles, questions, and options;

[0019] S2: Input the data in the training set into the embedding layer to extract features, and obtain the article representation, question representation and option representation;

[0020] S3: Input the article representation, question representation, and option representation into the encoding layer respectively to obtain the contextual information representation of the article, the contextual information representation of the question, and the contextual information representation of the option.

[0021] S4: An attention interaction layer is used to interact with the contextual information representations of the article, the question, and the options to obtain deep interactive information between the question, the article, and the answer.

[0022] S5: Input the deep interactive information of question-article-answer into the answer output layer to obtain the question result;

[0023] S6: Construct the model's loss function based on the problem results; adjust the model's parameters, and complete the model training when the loss function converges.

[0024] The model employs gated recurrent units to simultaneously aggregate local information of the text and global information of all word nodes in the entire sentence, including: the text representation at position i, and temporal information at a certain time step t, specifically represented as:

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] in, This represents the hidden state of the text at time step t with position i in the gated loop unit. W represents the state of the left-updating gate neuron at the current time step t. l U represents the trainable parameters of the first network. l V represents the trainable parameters for the second network. l b are the trainable parameters of the third network. l The first deviation parameter, W represents the state of the right-updating gate neuron at the current time step t. r U is the trainable parameter for the fourth network. r V represents the trainable parameters of the fifth network. r b are the trainable parameters of the sixth network. r This is the second deviation parameter. W represents the state of the update gate neuron at the current time step t. z For the trainable parameters of the seventh network, U z V represents the trainable parameters of the eighth network. z b are the trainable parameters of the ninth network. z The third deviation parameter is σ, where softmax() is the softmax function and σ is the sigmoid function. This is the concatenation of the hidden states at time step t-1 between the word at text position i and its two adjacent words, where [] represents the concatenation operation, and g t-1 For the hidden information of the sentence-level nodes at the previous time step t-1, x iIt is the feature representation of the word at text position i. represents the state of the memory gate neuron at the current time step t, and ⊙ represents point multiplication.

[0031] The state of the memory gate neurons at the current time step t of the model Specifically, it is expressed as follows:

[0032]

[0033]

[0034]

[0035] Where W represents the trainable parameters of the tenth network, U represents the trainable parameters of the eleventh network, V represents the trainable parameters of the twelfth network, b represents the fourth bias parameter, tanh() represents the tanh function, and f i t W represents the reset gate neuron state at the current time step t. f For the trainable parameters of the thirteenth network, U f V represents the trainable parameters of the fourteenth network. f b is the trainable parameter for the fifteenth network. f This is the fifth deviation parameter.

[0036] The hidden information g of the sentence-level nodes at time step t of the model t Specifically, it is expressed as:

[0037]

[0038]

[0039]

[0040]

[0041]

[0042] in, This represents the hidden state at time step t-1, where the text position is i. g represents the global information at the current time step t. t Update the state of the gate neuron, W g U is the trainable parameter for the sixteenth network. g V represents the trainable parameters of the seventeenth network. g b is the trainable parameter for the eighteenth network. g This is the sixth deviation parameter. This indicates the globally hidden state at the current time step t. W represents the hidden state update gate neuron state at the current time step t. u For the trainable parameters of the nineteenth network, U u V is the trainable parameter for the twentieth network. u For the trainable parameters of the twenty-first network, b u This is the seventh deviation parameter. g represents the global information at the current time step t. t The memory gate neuron states are defined by W1 (trainable parameters of the 22nd network), U1 (trainable parameters of the 23rd network), V1 (trainable parameters of the 24th network), and b1 (bias parameter of the 8th network). g represents the global information at the current time step t. t Reset the gate neuron state; W2 represents the trainable parameters of the 25th network, U2 represents the trainable parameters of the 26th network, and V2 represents the trainable parameters of the 27th network. f is the ninth deviation parameter, softmax() is the softmax function, σ is the sigmoid function, ⊙ represents dot product, and tanh() is the tanh function.

[0043] The global hidden state at the current time step t is specifically represented as follows:

[0044]

[0045]

[0046] in, For word x at time t-1 i Importance in the text W represents the hidden state at time step (t-1) with text position i. w For the 28th network trainable parameters, x i It is the feature representation of the word at text position i.

[0047] The model employs a question-article-answer three-attention mechanism, explicitly involving the contextual information of the article in the calculation of attention weights, thereby obtaining the deep interaction information H between the question, article, and answer. attention The deep interaction information between the question, article, and answer is as follows:

[0048] H attention =[H q H a ]

[0049] Among them, H q Representing the question q ′ As the query value, the answer is a ′ H serves as a deep interactive information between the question and answer, acting as both the value and the key. aThe answer is a. ′ As a query value, question q ′ As a deep interactive information between the answer and the key, [] represents vector concatenation.

[0050] The calculation formula for the question-article-answer three-attention mechanism includes:

[0051]

[0052]

[0053]

[0054] F9Query,Key i ,p j ′ ) = softmax(r T tanh(W3Query+U3Key i +V3p j ′ ))

[0055] in, Value i value and p j ′ Value fusion information, Value i p represents the contextual information at the i-th position of the Value. j ′ F(Query, Key) represents the contextual information of the article at position j. i ,p j ′ This indicates calculating the similarity between the Query, Key, and the contextual information of the article. i This represents the contextual information at the i-th position of the Key value. W3 is the trainable parameter of the 28th network, U3 is the trainable parameter of the 29th network, V3 is the trainable parameter of the 30th network, and r T It is the transpose of the thirty-first trainable parameter, softmax() is the softmax function, and tanh() is the tanh function; This indicates that the calculated similarity is normalized, where lenk represents the length of the key value, l represents the length of the article, lenv represents the length of the value value, and H represents the output of the question-article-answer three-attention mechanism.

[0056] The loss function expression for the model is:

[0057]

[0058] Where N is the sample size used to calculate the average, y ic The sign function is p, which takes the value 1 when the true class of sample i equals c, and 0 otherwise. ic Let be the probability that observed sample i belongs to c.

[0059] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A question and answer method based on a gated global attention machine reading comprehension model, characterized in that, The method comprises the following steps: acquiring an article to be answered, inputting the article to be answered into a trained gated global attention machine reading comprehension model, and obtaining a question result; The gated global attention machine reading comprehension model comprises an embedding layer, an encoding layer, an attention interaction layer, and an answer output layer; The process of training the gated global attention machine reading comprehension model comprises the following steps: S1: acquiring a training data set, which comprises an article, a question, and an option; S2: inputting the data in the training set into the embedding layer for feature extraction to obtain article representation, question representation, and option representation; S3: inputting the article representation, question representation, and option representation into the encoding layer respectively to obtain context information representation of the article, context information representation of the question, and context information representation of the option; S4: using the attention interaction layer to perform information interaction on the context information representation of the article, the context information representation of the question, and the context information representation of the option to obtain deep interaction information of the question-article-answer; The attention interaction layer is a question-article-answer three-attention mechanism; the context information representation of the article, the context information representation of the question, and the context information representation of the option are interacted through the question-article-answer three-attention mechanism; and the deep interaction information of the question-article-answer is: ； wherein, represents the question as the Query value, the answer as the Value value and the Key value, the deep interaction information of question-answer, represents the answer as the Query value, the question as the Value value and the Key value, the deep interaction information of answer-question, [] represents vector splicing; The question-article-answer three-attention mechanism is: ； ； ； ； wherein, represents the fusion information of the value and the value, represents the context information representation at the i-th position of the value, represents the context information representation at the j-th position of the article, represents the similarity of the query, the key and the context information representation of the article, represents the context information representation at the i-th position of the key, is the twenty-eighth network trainable parameter, is the twenty-ninth network trainable parameter, is the thirtieth network trainable parameter, is the transpose of the thirty-first trainable parameter, and softmax() is the softmax function, is the tanh function; represents the normalization of the calculated similarity, represents the length of the key, represents the length of the article, represents the length of the value, represents the output of the question-article-answer triple attention mechanism; S5: inputting the deep interaction information of the question-article-answer into the answer output layer to obtain a question result; S6: constructing a loss function of the model according to the question result; adjusting the parameters of the model, and completing the training of the model when the loss function converges.

2. The method according to claim 1, wherein, The encoding layer is a gated recurrent unit, and the process of processing the article representation, question representation, and option representation by the encoding layer comprises the following steps: wherein, denotes the hidden state of the text position i in the gating recurrent unit at the t-th time step, denotes the left update gate neuron state at the current time step t, is a first network trainable parameter, is a second network trainable parameter, is a third network trainable parameter, is a first bias parameter, denotes the right update gate neuron state at the current time step t, is a fourth network trainable parameter, is a fifth network trainable parameter, is a sixth network trainable parameter, is a second bias parameter, denotes the update gate neuron state at the current time step t, is a seventh network trainable parameter, is an eighth network trainable parameter, is a ninth network trainable parameter, is a third bias parameter, and softmax() is a softmax function, is a sigmoid function, is a concatenation of the word at the text position i with its two neighboring words and the previous t-1 time step hidden state of the word at the text position i, and [] is a concatenation operation, is the hidden information of the node at the sentence level at the previous t-1 time step, is a word feature representation of the word at the text position i, denotes the memory gate neuron state at the current time step t, denotes a point-wise multiplication.

3. The method according to claim 2, wherein, The calculation formula of the memory gate neuron state of the current time step t is: wherein, is a tenth network trainable parameter, is an eleventh network trainable parameter, is a twelfth network trainable parameter, b is a fourth bias parameter, is a tanh function, denotes the reset gate neuron state at the current time step t, is a thirteenth network trainable parameter, is a fourteenth network trainable parameter, is a fifteenth network trainable parameter, is a fifth bias parameter.

4. The method according to claim 2, wherein, hidden information of the node at the sentence level at the t-th time step The expression is: wherein, denotes the hidden state of the text position i at the t-1 time step, denotes the global information at the current time step t updates the gate neuron state, is a sixteenth network trainable parameter, is a seventeenth network trainable parameter, is an eighteenth network trainable parameter, is a sixth bias parameter, denotes the global hidden state at the current time step t, denotes the hidden state update gate neuron state at the current time step t, is a nineteenth network trainable parameter, is a twentieth network trainable parameter, is a twenty-first network trainable parameter, is a seventh bias parameter, denotes the global information at the current time step t updates the memory gate neuron state, is a twenty-second network trainable parameter, is a twenty-third network trainable parameter, is a twenty-fourth network trainable parameter, is an eighth bias parameter, denotes the global information at the current time step t updates the reset gate neuron state, is a twenty-fifth network trainable parameter, is a twenty-sixth network trainable parameter, is a twenty-seventh network trainable parameter, is a ninth bias parameter, and softmax() is a softmax function, is a sigmoid function, denotes a point-wise multiplication, is a tanh function.

5. The method according to claim 4, wherein, The expression of the global hidden state of the current time step t is: wherein, is the t-1 time word importance in the text, denotes the hidden state at the t-1 time step for text position i, is the twenty-eighth network trainable parameter, is a word feature representation for text position i.

Images