Knowledge distillation method and device

Abstract

The present invention discloses a knowledge distillation method and device, wherein, a knowledge distillation method includes: in the speaker embedding learning task, input the audio data of the same speaker into the teacher model and the student model, wherein the teacher model and the student model are both Including speaker embedding extraction and speaker posterior probability distribution prediction; taking the speaker embedding extraction of the teacher model as the standard, the gap between the speaker embedding extraction of the student model and the speaker embedding extraction of the teacher model is limited to the first preset range In order to optimize the student model; or take the speaker posterior probability distribution prediction of the teacher model as the standard, limit the gap between the speaker posterior probability distribution prediction of the student model and the teacher speaker posterior probability distribution within the second To optimize the student model within a preset range; use the optimized student model for deployment and / or prediction. In this way, the small model can be trained by the large model with good performance, and then the small model can be deployed and used.

Description

technical field [0001] The invention belongs to the technical field of speech data processing, and in particular relates to a knowledge distillation method and device. Background technique [0002] In related technologies, i-vector is a classic speaker embedding learning method, which is based on a traditional factor analysis model, and essentially obtains a low-dimensional space representation of a Gaussian supervector. [0003] Speaker embedding learning based on deep neural networks first trains a network whose goal is to distinguish different speakers, and then extracts speaker embedding representations from a specific layer (embedding extraction layer). A large number of papers prove that large networks and deep networks usually achieve better results. [0004] Deep speaker embedding learning is a very effective method for speaker identity modeling. Ultra-deep models such as residual networks have achieved good performance, but for real application scenarios with limit...

AI Technical Summary

Problems solved by technology

[0006] i-vector can be regarded as a single-layer linear model in essence. It is difficult to perform robust modeling for complex data, and the performance of short-term data is not very good.

[0007] For practical application scenarios, ultra-deep networks (such as residual networks) are difficult to deploy due to the huge amount of calculations, and simple shallow models with small parameters often fail to meet performance requirements

Images