Knowledge distillation method and device

Abstract

The invention discloses a knowledge distillation method and device. The knowledge distillation method comprises the steps that when a speaker is in a learning task, audio data of the speaker is inputinto a teacher model and a student model, wherein the teacher model and the student model involve speaker embedding extraction and speaker posterior probability distribution prediction respectively; the speaker embedding extraction of the teacher model is taken as the standard, and the difference between the speaker embedding extraction of the student model and the speaker embedding extraction ofthe teacher model is limited in a first preset range to optimize the student model; or the speaker posterior probability distribution prediction of the teacher model is taken as the standard, and thedifference between the speaker posterior probability distribution prediction of the student model and the speaker posterior probability distribution prediction of the teacher model is limited in a second preset range to optimize the student model; the optimized student model is used for deployment and / or prediction. Therefore a small model can be deployed and used by training the small model through a large model with good performance.

Description

technical field [0001] The invention belongs to the technical field of speech data processing, and in particular relates to a knowledge distillation method and device. Background technique [0002] In related technologies, i-vector is a classic speaker embedding learning method, which is based on a traditional factor analysis model, and essentially obtains a low-dimensional space representation of a Gaussian supervector. [0003] Speaker embedding learning based on deep neural networks first trains a network whose goal is to distinguish different speakers, and then extracts speaker embedding representations from a specific layer (embedding extraction layer). A large number of papers prove that large networks and deep networks usually achieve better results. [0004] Deep speaker embedding learning is a very effective method for speaker identity modeling. Ultra-deep models such as residual networks have achieved good performance, but for real application scenarios with limit...

AI Technical Summary

Problems solved by technology

[0006] i-vector can be regarded as a single-layer linear model in essence. It is difficult to perform robust modeling for complex data, and the performance of short-term data is not very good.

[0007] For practical application scenarios, ultra-deep networks (such as residual networks) are difficult to deploy due to the huge amount of calculations, and simple shallow models with small parameters often fail to meet performance requirements

Images