Language model training method and system in self-reconstruction mode and computer readable medium

Abstract

The invention relates to the technical field of language processing, in particular to a language model training method in a self-reconstruction mode, which comprises the following steps of: S1, extracting at least one sentence to be trained from a pre-training text, segmenting the sentence to be trained into single word sequences, and mapping corresponding single sub-sequences into a text matrix through position coding; s2, establishing a neural network structure in combination with a transformer model and a self-attention mechanism; s3, taking the text matrix as an input sample of a neural network structure, and taking the transformer model as a parameter to train and optimize to obtain a target function; and S4, repeating the steps S1 to S3 to update the target function until a set optimization condition is reached so as to obtain a pre-training model. The invention also provides a system and a computer readable medium.

Description

【Technical field】 [0001] The present invention relates to the technical field of language processing, in particular to a language model training method, system and computer-readable medium in a self-reconstruction mode. 【Background technique】 [0002] At present, the most advanced pre-training language models are divided into two categories, namely autoregressive language model (Autoregressive Model) and autoencoding language model (Autoencoding Model); GPT and GPT2 are autoregressive language models with better performance. The training goal of the autoregressive model is to correctly guess the next word based on the previous text. BERT is a representative self-encoding language model. The training goal of BERT is to correctly infer the masked or replaced words according to the context. [0003] The above two types of pre-trained language models use the Transformer (conversion) model. The model combines AttentionEncoder (attention encoder) and Attention Decoder (attentio...

AI Technical Summary

Problems solved by technology

[0007] Aiming at the defects of low prediction accuracy and high cost of existing language models, the present invention provides a self-reconstruction language model training method, system and computer-readable medium

Images