Unlock instant, AI-driven research and patent intelligence for your innovation.

Cyclic neural network-based training method for Mongolian language models

A technology of recurrent neural network and language model, applied in the training field of Mongolian language model, can solve the problems of lack of semantic information injection, lack of long-distance information description ability, etc.

Active Publication Date: 2018-09-18
INNER MONGOLIA UNIV OF TECH
View PDF7 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Based on the original N-Gram language model, various models such as NNLM and RNNLM have been gradually developed. The main problems of existing language models are the lack of ability to describe long-distance information and the lack of semantic information injection.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Cyclic neural network-based training method for Mongolian language models
  • Cyclic neural network-based training method for Mongolian language models
  • Cyclic neural network-based training method for Mongolian language models

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030] 1. Model construction

[0031] The MLMRNN model structure includes an input layer, a hidden layer and an output layer. The input layer is x(t), the hidden layer is s(t), and the output layer is y(t). At time t, the input vector x(t) includes three parts, namely w(t), s(t-1) and f(t), where w(t) represents the one-hot word vector of the word input at time t Indicates; s(t-1) is the output of the hidden layer at time t-1; f(t) is the context word vector trained by Skip-Gram at time t, and its dimension is much smaller than |V|; the hidden layer is s(t ); the output vector is represented by y(t), which includes two parts, one part is a category layer neuron, and the other part is a Mongolian word neuron, c(t) is the category layer of the clustering of word vectors carried out on the vocabulary, output The vector y(t) represents the probability of the next word w(t+1).

[0032] In the network, U, W, and F are the weight matrix between the input layer and the hidden layer,...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a DNN (Deep Neural Network)-based Mongolian acoustic model on the basis of a conventional N-Gram language model, an NNLM (Neural Network Language Model) and an RNNLM (RecurrentNeural Network Language Model) and discloses a training method of the Mongolian acoustic model. According to the method provided by the invention, a context word vector and a word vector containing semantic information categories are introduced into an input layer to ensure that longer-distance historical information can be learned and relevant semantic category information is also injected at thesame time, so that the problems mainly existing in the conventional language models are effectively solved.

Description

technical field [0001] The invention belongs to the field of Mongolian speech recognition, and in particular relates to a training method of a Mongolian language model based on a recurrent neural network. Background technique [0002] Language models are widely used in natural language processing, such as speech recognition, machine translation, question answering systems and other applications. The language model is modeled by identifying the prior probability of word sequences that are allowed to appear in the language, and provides grammatical and syntactic constraints for word sequences. Based on the original N-Gram language model, various models such as NNLM and RNNLM have been gradually developed. The main problems of existing language models are the lack of ability to describe long-distance information and the lack of semantic information injection. Contents of the invention [0003] Since the language model based on the cyclic neural network can better avoid the d...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06K9/62G06N3/04
CPCG06N3/045G06F18/214G06F18/2415
Inventor 马杰马志强杨瑞
Owner INNER MONGOLIA UNIV OF TECH