Character-level language model prediction method based on local perceptual recurrent neural network

A recursive neural network and language model technology, applied in biological neural network models, neural learning methods, etc., can solve the problems of difficult division of functions, single way of data and information inflow, and difficulty in determining the number of hierarchical neurons. Accuracy, the effect of strong information integration ability

Inactive Publication Date: 2018-10-09
HOHAI UNIV
View PDF3 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, the forms and functions of each layer of the traditional multi-layer recurrent neural network are similar, which makes it difficult to divide the functions of each layer of recurrent neural network, and it is not easy to determine the required number of layers and the number of neurons in...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Character-level language model prediction method based on local perceptual recurrent neural network
  • Character-level language model prediction method based on local perceptual recurrent neural network
  • Character-level language model prediction method based on local perceptual recurrent neural network

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0038] The technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0039] Such as Figures 1 to 3 As shown, this embodiment provides a character-level language model prediction method based on a local perception recurrent neural network, which specifically includes the following steps:

[0040] S1: Obtain the original data, preprocess the data, divide the data into training data, test data and verification data, use the data set PTB to train the model, the training set has 5.2M characters and 900K words; the verification set has 400K characters and ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a character-level language model prediction method based on the local perceptual recurrent neural network. A processing mode of the recurrent neural network is utilized, threelayers of networks are combined together in layers, the low layer obtains features among local characters, the high layer obtains semantic features of texts, so a new model is made to have the stronger information integration capability, and processing longer data sequences is easier. The method is advantaged in that the monitored BPTT-RNN (Recurrent Neural Network Reverse Propagation Algorithm) method is utilized for training; firstly, adadalta (Adaptive Learning Rate Adjustment) optimization training is utilized to a test set, BPC is lower than 1.45, rapid convergence is carried out, the SGD(Stochastic Gradient Descent) optimization method with the learning rate of 0.0001 and a momentum of 0.9 is utilized for training, and the better test result is obtained.

Description

technical field [0001] The invention belongs to the field of natural language processing, in particular to a character-level language model prediction method based on a local perceptual recursive neural network. Background technique [0002] Recurrent neural networks are a dynamic model with great expressive power, because RNN has high-dimensional hidden nonlinear internal state, which enables it to extract a priori dependency information from previously processed information. In theory, a RNN with a sufficiently large hidden state can generate sequences of arbitrary complexity, and it has been proven that RNNs are Turing-complete given any number of hidden neurons; but in practice, standard RNNs cannot store longer The existing input sequence information, so although the ability of RNN is very attractive to people, the internal hidden state becomes unstable after repeated recursive processing, and the gradient is easy to disappear or expand. This limits the application of ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06N3/02G06N3/08
CPCG06N3/02G06N3/08
Inventor 刘惠义王刚陶颖
Owner HOHAI UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products