Character-level language model prediction method based on local perceptual recurrent neural network

Abstract

The invention discloses a character-level language model prediction method based on the local perceptual recurrent neural network. A processing mode of the recurrent neural network is utilized, threelayers of networks are combined together in layers, the low layer obtains features among local characters, the high layer obtains semantic features of texts, so a new model is made to have the stronger information integration capability, and processing longer data sequences is easier. The method is advantaged in that the monitored BPTT-RNN (Recurrent Neural Network Reverse Propagation Algorithm) method is utilized for training; firstly, adadalta (Adaptive Learning Rate Adjustment) optimization training is utilized to a test set, BPC is lower than 1.45, rapid convergence is carried out, the SGD(Stochastic Gradient Descent) optimization method with the learning rate of 0.0001 and a momentum of 0.9 is utilized for training, and the better test result is obtained.

Description

technical field [0001] The invention belongs to the field of natural language processing, in particular to a character-level language model prediction method based on a local perceptual recursive neural network. Background technique [0002] Recurrent neural networks are a dynamic model with great expressive power, because RNN has high-dimensional hidden nonlinear internal state, which enables it to extract a priori dependency information from previously processed information. In theory, a RNN with a sufficiently large hidden state can generate sequences of arbitrary complexity, and it has been proven that RNNs are Turing-complete given any number of hidden neurons; but in practice, standard RNNs cannot store longer The existing input sequence information, so although the ability of RNN is very attractive to people, the internal hidden state becomes unstable after repeated recursive processing, and the gradient is easy to disappear or expand. This limits the application of ...

AI Technical Summary

Problems solved by technology

[0004] However, the forms and functions of each layer of the traditional multi-layer recurrent neural network are similar, which makes it difficult to divide the functions of each layer of recurrent neural network, and it is not easy to determine the required number of layers and the number of neurons in each layer; and When data is input into the traditional multi-layer recursive neural network, each time step, the data is simply transmitted from the underlying neural network to the upper layer neural network, and the data information flows in a single way, making it difficult to process long data sequences

Images