Low-Resource Commonsense Question Answering Method Based on Prompt Learning and Fine-Grained Contrastive Learning

Abstract

The present invention discloses a low-resource common sense question answering method based on prompt learning and fine-grained contrastive learning, belonging to the field of natural language processing. The method includes: converting the cloze-style format based on prompt learning; performing contrast on options based on fine-grained; predicting the correct answer. Compared with the prior art, the present invention introduces prompt learning and fine-grained-based contrastive learning, which can reduce redundancy during model training under low-resource conditions and significantly improve the accuracy of the model.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and more particularly, to a low-resource common sense question answering method based on prompt learning and fine-grained contrastive learning for common sense question answering in low-resource scenarios. Background Art

[0002] Common sense question answering is a type of question answering that relies on implicit common sense knowledge that is not present in the context. For example, when the word "sun" is mentioned in a question, people cannot directly know that the sun is a star and has a very high temperature. Common sense question answering is to use a lot of such implicit common sense to select the correct answer to the question.

[0003] Some researchers have used graph neural networks (GNNs) to better utilize the knowledge in knowledge bases and pre-trained language models. There are also works that use additional knowledge bases (such as ConceptNet, ATOMIC) to help answer common sense questions. Recently, several methods have studied generating common sense question pairs from additional knowledge bases and then using the generated data to fine-tune pre-trained language models. However, due to the limited number of predefined relationships and the sparsity of knowledge bases, many entity pairs cannot find their corresponding relationships in the knowledge base, and some entities cannot even be found in the knowledge base. In addition, the cost of constructing a knowledge base is very high.

[0004] With the explosion of the parameter scale of pre-trained language models, many works have focused on studying the use of implicit common sense knowledge inside the models. Some researchers have also used the T5 model to generate clues for inferring the question background. In low-resource situations, there are works that design prompt templates to better obtain knowledge from pre-trained language models. Although these pre-trained models contain a large amount of knowledge, the large-scale parameters make the models suffer from redundancy, especially in low-resource scenarios.

[0005] Therefore, how to reduce redundancy in low-resource scenarios to better utilize the implicit common sense knowledge of pre-trained models is an urgent research topic. Summary of the Invention

[0006] The object of the present invention is to provide a low-resource common sense question answering method based on prompt learning and fine-grained contrastive learning, which can better utilize the built-in knowledge in pre-trained models and alleviate model redundancy in low-resource situations.

[0007] The specific technical solution for achieving the object of the present invention is as follows:

[0008] A low-resource common sense question answering method based on prompt learning and fine-grained contrastive learning, characterized in that the method includes: Definition of question-answer pairs: The j-th question-answer pair where |D| is the number of samples, q (j)is the question of the j-th question-answer pair, O (j) is the set of options for the j-th question-answer pair; where |O (j) | is the number of options, represents the i-th option of the j-th question-answer pair; the goal of the question-answer is to find the correct option corresponding to the question q (j) from the option set O (j) ;

[0009] Cloze-style format conversion based on prompt learning: The original question q (j) is converted into a cloze-style question c (j) through a rule-based syntactic rewriting method, where the position of the option is represented by the [OPTION] token, and then the [OPTION] token is replaced with the option and a cloze-style question with specific options is obtained Then, based on the rules of prompt learning rewrite into a rewritten sentence based on prompt learning The specific rewriting rules are as follows:

[0010]

[0011] Fine-grained comparison of options: Each training will have a batch of question-answer pairs Through the above conversion, a set of sentences rewritten based on prompt learning is obtained Then, the set of sentences S is encoded by the pre-trained language model albert-xxlarge-v2 to obtain the representation of the [MASK] token where d represents the representation dimension, and n represents the number of options for a question; then randomly select the [MASK] token representations of two options, denoted as Calculate the correlation matrix to measure the similarity of different dimensions between two options, M kt represents the similarity between dimension k and dimension t, and the correlation matrix M kt is calculated as follows:

[0012]

[0013] where represents the value of the k-th dimension of the b-th question pair of the first option among the two randomly selected options in this batch of question-answer pairs, similarly.

[0014] The first training task: Through The loss function makes the values of the diagonal elements of the correlation matrix M close to 1 and the values of the non-diagonal elements close to 0, that is, the similarity of the same dimension is close to 1, and the similarity of different dimensions is close to 0. In this way, the information represented between different dimensions can be made different, thereby reducing redundancy and improving the accuracy of the model; in the formula, α is a hyperparameter used to balance the loss value of the diagonal and the loss value of the non-diagonal.

[0015]

[0016] Predict the correct answer: Rewrite the sentence Obtain the probability of each token t appearing at the [MASK] token through the pre-trained language model MLM:

[0017]

[0018] Each option is regarded as a binary classification problem, so the score of the option as the answer is determined by the probabilities of the "yes" and "no" tokens appearing at the [MASK] token is measured, and the option with the highest score will be considered the correct answer;

[0019]

[0020] The second training task: Through make the score of the correct option higher and the score of the wrong option lower;

[0021]

[0022] Among them is 's label; when is the correct option is 1, otherwise is 0; Cb represents the cross-entropy loss function.

[0023] The beneficial effects of the present invention:

[0024] Prompt learning improves the utilization rate of the implicit common sense knowledge inside the pre-trained model in the present invention, and can better utilize the internal knowledge of the model in low-resource situations; fine-grained contrast learning enables the present invention to significantly reduce model redundancy in low-resource situations, thereby improving the accuracy of predicting answers. Description of the drawings

[0025] Figure 1 is the flowchart of the present invention. Detailed implementation manners

[0026] In combination with the following specific embodiments and the accompanying drawings, the present invention will be further described in detail. The processes, conditions, experimental methods, etc. for implementing the present invention, except for the specifically mentioned content below, are all common knowledge and general knowledge in the art, and the present invention has no particularly restricted content.

[0027] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will further describe the embodiments of the present invention in detail with reference to the examples in the CommonsenseQA dataset in combination with the accompanying drawings.

[0028] Embodiment

[0029] Definition of question-answer pair: The j-th question-answer pair where |D| is the number of samples, q (j) is the question of the j-th question-answer pair, and O (j) is the option set of the j-th question-answer pair; where |O (j) | is the number of options, represents the i-th option of the j-th question-answer pair; the goal of the question-answer is to find the correct option corresponding to the question q (j) from the option set O (j) ; here, the example question is "Where did Puppigerus live?", and the options for the example question are "bog", "ocean", "land", "marsk" respectively.

[0030] Cloze-style format conversion based on prompt learning: The original question q (j) is converted into a cloze-style question c (j) through a rule-based syntactic rewriting method. The example question will be rewritten from "Where did Puppigerus live?" to the example cloze-style question "Puppigerus live at [OPTION]", where the position of the option will be represented by the [OPTION] token. Then, the [OPTION] token is replaced with a specific option and a cloze-style question containing specific options is obtained The example cloze-style question containing specific option 1 is "Puppigerus live at bog", and then based on the rules of prompt learning rewrite into a rewritten sentence based on prompt learning The rewrite of the example cloze-style question containing specific option 1 will become "Puppigerus live at bog, right? [MASK]", and the specific rewrite rules are as follows:

[0031]

[0032] Compare options based on fine-grained: There will be a batch of question-answer pairs during each training. Through the above transformation, a set of sentences rewritten based on prompt learning is obtained. Then, the set of sentences S is encoded by the pre-trained language model albert-xxlarge-v2 to obtain the representations of [MASK] tokens. Where d represents the representation dimension, and n represents the number of options for a question; then randomly select the [MASK] token representations of two options, denoted as By calculating the correlation matrix to measure the similarity of different dimensions between two options, M kt represents the similarity between dimension k and dimension t, and the correlation matrix M kt is calculated as follows:

[0033]

[0034] Where represents the value of the k-th dimension of the b-th question-answer pair of the first option among the two randomly selected options in this batch of question-answer pairs. Similarly.

[0035] The first training task: Through the loss function makes the values of the diagonal elements of the correlation matrix M close to 1, and the values of the non-diagonal elements close to 0, that is, the similarity of the same dimension is close to 1, and the similarity of different dimensions is close to 0. In this way, the information represented between different dimensions can be made different, thereby reducing redundancy and improving the accuracy of the model; α in the formula is a hyperparameter used to balance the loss value of the diagonal and the loss value of the non-diagonal.

[0036]

[0037] Predict the correct answer: Rewrite the sentence through the pre-trained language model MLM to obtain the probability of each token t at the position of the [MASK] token:

[0038]

[0039] Each option is regarded as a binary classification problem, so the scores of the options as answers are measured by the probabilities of the "yes" and "no" tokens at the position of the [MASK] token. The option with the highest score will be considered the correct answer.

[0040]

[0041] The second training task: Through Make the score of the correct option higher and the score of the wrong option lower;

[0042]

[0043] where is the label of; when is the correct option is 1, otherwise is 0; CE represents the cross-entropy loss function.

Claims

1. A low-resource commonsense question answering method based on prompt learning and fine-grained contrastive learning, characterized in that, The method includes: Definition of question-answer pairs: The j-th question-answer pair where |D| is the number of samples, q (j) is the question of the j-th question-answer pair, and O (j) is the set of options for the j-th question-answer pair; where |O (j) | is the number of options, represents the i-th option of the j-th question-answer pair; The goal of the question-answer is to find the correct option corresponding to the question q (j) from the set of options O (j) for the question q Cloze-style format transformation based on prompt learning: transforming the original question q (j) into a cloze-style question c through a rule-based syntactic rewriting method (j) , where the position of the option is represented by the [OPTION] token, and then replacing the [OPTION] token with the option to obtain a cloze-style question with specific options Then, based on the rules of prompt learning rewrite into a rewritten sentence based on prompt learning The specific rewriting rules are as follows: Compare options based on fine-grained: There will be a batch of question-answer pairs during each training Through the above transformation, a set of sentences rewritten based on prompt learning is obtained Then, the set of sentences S is encoded by the pre-trained language model albert-xxlarge-v2 to obtain the representations of [MASK] tokens where d represents the representation dimension and n represents the number of options for a question; then, randomly select the [MASK] token representations of two options, denoted as By calculating the correlation matrix to measure the similarity of different dimensions between two options, M kt represents the similarity between dimension k and dimension t, and the correlation matrix M kt is calculated as follows: wherein represents the value of the k-th dimension of the b-th Q&A pair of the first option among the two options randomly selected from this batch of Q&A pairs, similarly; The first training task: By using the loss function, the values of the diagonal elements of the correlation matrix M are made close to 1, and the values of the non-diagonal elements are made close to 0. That is, the similarity of the same dimension is close to 1, and the similarity of different dimensions is close to 0. In this way, the information represented between different dimensions can be made different, thereby reducing redundancy and improving the accuracy of the model. In the formula, α is a hyperparameter used to balance the loss value of the diagonal and the loss value of the non-diagonal. Predicted correct answer: the rewritten sentence The probability of each token t occurring at the [MASK] token is obtained through the pre-trained language model MLM: Each option is regarded as a binary classification problem, so the score of the option as the answer is measured by the probabilities of the yes and no tokens occurring at the [MASK] token, and the option with the highest score will be considered the correct answer; ​ Second training task: By making the scores of correct options higher and the scores of incorrect options lower; Among them is 's label; when is the correct option is 1, otherwise is 0; CE represents the cross-entropy loss function.

Images