A distillation-based static vector model generation method and system

Abstract

This invention discloses a static vector model generation method based on distillation, comprising: obtaining all independent lexical units in the basic model's word segmenter; inputting each independent lexical unit individually into the basic model for forward propagation to extract representations and constructing an initial embedding matrix; applying principal component analysis to the initial embedding matrix for dimensionality reduction; and using a smooth inverse frequency weighting algorithm based on Zipf's law, employing vocabulary ranking as a proxy variable to calculate lexical probabilities, and performing weighted optimization on the dimensionality-reduced embedding matrix to generate the target static vector model. This invention eliminates the dependence of traditional model distillation on massive amounts of data and GPUs, simplifies complex Transformer nonlinear inference to an extremely fast table lookup operation through computational collapse, and retains most of the deep semantic understanding capabilities of the original model while reducing model size and increasing CPU inference speed.

Images