Language model pre-training method for common sense concept enhancement

Abstract

The invention discloses a common sense concept enhanced language model pre-training method, which comprises the following steps: step 1) collecting a corpus set, and constructing an unsupervised corpus set based on common sense concepts, the unsupervised corpus set comprising a plurality of sentences, each sentence comprising a plurality of common sense concepts and the position of each common sense concept in the sentence; 2) based on the unsupervised corpus set, randomly covering the common sense concept to form a training sample, inputting the training sample into a pre-established language model for training, and obtaining a pre-training language model with enhanced common sense concept by predicting the covered common sense concept; and step 3) obtaining a prediction sequence of language modeling through the pre-training language model enhanced by the common sense concept. According to the method, the common sense understanding ability of the pre-training language model is effectively enhanced, and experiments prove that the question and answer accuracy can be obviously improved by finely adjusting the common sense concept enhanced pre-training language model on the common sense question and answer task.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a language model pre-training method for enhancing common sense concepts. Background technique [0002] In recent years, pre-training language model technology has achieved rapid development in the field of natural language processing, and has formed a new generation of natural language processing paradigm represented by pre-training-fine-tuning. The technique aims to perform unsupervised pre-training by using a masked language modeling (MLM) loss function on a large-scale unlabeled corpus, followed by supervised fine-tuning on a small-scale labeled dataset. Taking "Situation: I am wearing a wristwatch." as an example, after randomly covering some words, the input sequence "Situation: [MASK]am wearing [MASK] wrist watch." can be obtained, and the pre-training goal of the model is to predict The masked words are "I" and "a". At present, the method of fine-tun...

AI Technical Summary

Problems solved by technology

The purpose of the present invention is to overcome the problem that the existing pre-training language model cannot display and model common sense concept information, and propose a language model pre-training method for common sense concept enhancement, through which the pre-training language model can be significantly improved common sense comprehension ability, and enhance its performance on downstream common sense question answering tasks

Images