Language model training method and prediction method

Abstract

The invention provides a language model training method and a prediction method. The training method comprises the following steps of: obtaining training text sequences, randomly generating a target position of a character or a word which needs to be modeled and analyzed through a language model in the training text sequence; carrying out mask replacement on characters or words at a target position; generating a text sequence subjected to mask replacement, generating a limited word list of replaced characters or words, generating probability distribution on a limited word list space accordingto the text sequence subjected to mask replacement and the limited word list, calculating a cross entropy function according to the replaced characters or words and the probability distribution on thelimited word list space, and carrying out iterative optimization. According to the method, the limited word list is introduced to the decoding end of the model, and the original word information is fully utilized during model training, so that confusable words are easier to distinguish by the language model, and the effect of the language model on tasks such as error detection or error correctionis improved.

Description

technical field [0001] The present application relates to the field of computer technology, in particular to a language model training method and prediction method. Background technique [0002] At present, the decoding candidate set of the decoding end of the language model based on Recurrent Neural Network (RNN) is the entire vocabulary space (for example, based on the Chinese character dictionary / word segmentation dictionary), the decoding space of the language model is too large, resulting in model complexity High and difficult to converge, which reduces the degree of discrimination of confusing words and reduces the effect of language models on tasks such as error detection or error correction. Contents of the invention [0003] This application proposes a language model training method and prediction method, which is used to solve the problems in the related art that the RNN-based language model has a reduced degree of discrimination for confusing words and poor resu...

AI Technical Summary

Problems solved by technology

[0003] This application proposes a language model training method and prediction method, which is used to solve the problems in related technologies that the RNN-based language model can reduce the degree of discrimination of confusing words and have poor effects on tasks such as error detection or error correction.

Images