Neural network training acceleration method based on lattice-free maximum mutual information criterion
A neural network training and maximum mutual information technology, applied in the field of neural network training acceleration, can solve the problems of high output layer dimension and large training time ratio, and achieve the effect of simplifying the calculation structure, speeding up training, and reducing the training time ratio
Inactive Publication Date: 2018-10-09
INST OF ACOUSTICS CHINESE ACAD OF SCI +1
View PDF0 Cites 8 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0004] The purpose of the present invention is, in order to solve the training neural network based on LFMMI criterion, because LFMMI training has adopted double training criterion (LFMMI criterion and CE criterion), and output layer dimension is too high, thereby cause output layer forward calculation and backward calculation Technical issues that take up a large proportion of training time
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0016] A neural network training acceleration method based on the gridless maximum mutual information criterion according to the present invention will be described in detail below with reference to the drawings and embodiments.
[0017] The neural network training based on the LFMMI criterion uses two loss values when calculating the loss function, including the loss value calculated based on the LFMMI criterion and the loss value calculated based on the CE criterion. The neural network will weight and sum the loss values calculated based on LFMMI and CE as the loss value of the entire neural network. Each layer of the neural network contains three specific modules: a fully connected matrix, an activation function module, and an activation function value normalization module. Among them, the fully connected matrix refers to all the connections formed between each output node and each input node in a certain layer of the neural network, which acts on the vector output by t...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a neural network training acceleration method based on a lattice-free maximum mutual information criterion. According to the method, an original high-rank matrix module is divided into two low-rank matrix modules through low-rank transformation of the output end of a neural network CE criterion under the lattice-free maximum mutual information criterion (LFMMI criterion). The final dimension after sub-matrixes in the two divided low-rank matrix modules are multiplied is consistent with previous full connection matrix. Neural network training is carried out by using the modified low-dimensional sub-matrixes under the condition of guaranteeing the overall dimensions of the output end of the neural network CE criterion unchanged. Therefore, an operation structure is simplified, the proportion of training time occupied by forward calculation and backward calculation in the output layer of the neural network is obviously reduced, and the training of the neural networkis accelerated.
Description
technical field [0001] The invention belongs to the field of speech recognition, and in particular relates to a neural network training acceleration method based on the gridless maximum mutual information criterion. Background technique [0002] Language is a unique function of human beings, and it is the easiest and most effective tool for communicating and transmitting information between people. People's research on computer speech mainly includes the following aspects: speech coding, speech synthesis, speech recognition, speech enhancement, speaker recognition, etc. Among these studies, speech recognition is an extremely important part. After decades of development, speech recognition technology has penetrated into all aspects of our lives, and its application range covers smart homes, smart phone assistants, and national defense security. [0003] Speech recognition technology mainly includes three major aspects: acoustic model, language model and decoder. At this st...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06N3/08
CPCG06N3/08
Inventor 黎塔张震程高峰万辛颜永红
Owner INST OF ACOUSTICS CHINESE ACAD OF SCI