A language model pre-training method combined with word class information

Abstract

The present invention relates to the technical field of language processing, in particular to a language model pre-training method combined with word class information, which includes the following steps: S1, providing a pre-training model and a pre-training text; S2, extracting a character string and forming a vocabulary; S3 1. Extract two sentences as the training sentence and divide the training sentence into a single-word sequence; S4, match the character string in step S2 with the word in the single-word sequence, and mark the character string matched with the word in the single-word sequence ; S5. Covering or replacing a word with a preset ratio in the word sequence, and simultaneously inputting the covered or replaced training sentence and the marked character string into the pre-training model to train and optimize the pre-training model; S6. Steps S2-S5 are repeated until the pre-training model reaches the set optimization condition to obtain an optimized pre-training model. The language model pre-training method combined with class word information and the pre-training model provided by the present invention have better performance on multiple downstream tasks.

Description

【Technical field】 [0001] The invention relates to the technical field of language processing, in particular to a language model pre-training method combined with word class information. 【Background technique】 [0002] At present, the most advanced pre-training language models are divided into two categories, namely Autoregressive Model and Autoencoding Model. GPT and GPT2 are well-performing autoregressive language models. The training goal of the autoregressive model is to correctly guess the next word based on the previous text. BERT is a representative self-encoding language model. The training goal of BERT is to correctly infer the masked or replaced words according to the context. Both language models have advantages and disadvantages. Autoregressive models can only combine the preceding text, but cannot simultaneously combine contextual content to complete specific tasks. On the other hand, the self-encoding language model can use context information at the same t...

AI Technical Summary

Problems solved by technology

[0007] Aiming at the defects of low prediction accuracy and high cost of the existing language model, the present invention provides a language model pre-training method combined with word class information

Images