Multi-mechanism mixed recurrent neural network model compression method

Abstract

The invention discloses a multi-mechanism mixed recurrent neural network model compression method. The multi-mechanism mixed recurrent neural network model compression method comprises A, carrying outcirculant matrix restriction: restricting a part of parameter matrixes in the recurrent neural network into circulant matrixes, and updating a backward gradient propagation algorithm to enable the network to carry out batch training of the circulant matrixes, B, carrying out forward activation function approximation: replacing a non-linear activation function with a hardware-friendly linear function during the forward operation process, and keeping the backward gradient updating process unchanged; C, carrying out hybrid quantization: employing different quantification mechanisms for differentparameters according to the error tolerance difference between different parameters in the recurrent neural network; and D, employing a secondary training mechanism: dividing network model training into two phases including initial training and repeated training. Each phase places particular emphasis on a different model compression method, interaction between different model compression methodsis well avoided, and precision loss brought by the model compression method is reduced to the maximum extent. According to the invention, a plurality of model compression mechanisms are employed to compress the recurrent neural network model, model parameters can be greatly reduced, and the multi-mechanism mixed recurrent neural network model compression method is suitable for a memory-limited andlow-delay embedded system needing to use the recurrent neural network, and has good innovativeness and a good application prospect.

Description

technical field [0001] The invention relates to the technical field of computer and electronic information, in particular to a multi-mechanism mixed recursive neural network model compression method. Background technique [0002] Recurrent neural network has powerful nonlinear fitting ability, and its natural recursive structure is very suitable for modeling sequence data, such as text, speech and video. At present, the recurrent neural network model has achieved an effect or accuracy close to or even surpassing that of humans in the field of natural language processing, especially in speech recognition and machine translation; It also has broad application prospects. These technologies are necessary to realize intelligent human-computer interaction, but there are many problems in running recurrent neural network models on embedded devices. On the one hand, the recurrent neural network model needs to store a large number of parameters, and the amount of calculation is huge...

AI Technical Summary

Problems solved by technology

[0006] The technical problem to be solved by the present invention is to propose a variety of efficient model compression mechanisms and hybrid methods for recurrent neural networks that cannot adapt to the storage resources and computing capabilities of embedded systems, so that the application of recurrent neural networks in embedded systems becomes possible

Images