Cross-dimensional knowledge migration method for migrating knowledge from high-dimensional deep learning model to low dimension

Abstract

The invention discloses a cross-dimensional knowledge migration method for migrating knowledge from a high-dimensional deep learning model to a low dimension. The method comprises the following steps:1) expanding n-1-dimensional data x to form pseudo n-dimensional data y; 2) inputting y into a teacher network, and extracting n-dimensional features; 3) calculating the mean value of the n-dimensional features extracted by the teacher network in the nth dimension; 4) inputting x into a student network, and extracting n-1-dimensional features; 5) enabling the feature output of the student networkto approach the feature output of the teacher network through the constraint of a loss function; (6) repeating the steps (1)-(5) for set times until the difference between the feature output of the student network and the feature output of the teacher network is smaller than a preset threshold value; and (7) migrating to the dimension n-2 according to the methods in the steps (1)-(6) until the dimension m is reached. When only low-dimensional data exists or only low-dimensional data can be adopted to improve the algorithm speed, a low-dimensional model can be obtained by adopting the method,knowledge of a high-dimensional model is stored in the low-dimensional model, and the feature extraction capacity similar to that of the high-dimensional model is achieved.

Description

technical field [0001] The invention belongs to the field of transfer learning, and realizes transferring knowledge from a high-dimensional model to a low-dimensional model. The low-dimensional model obtained through the training of the present invention stores the knowledge of the high-dimensional model, and has the feature extraction capability similar to that of the high-dimensional model. Background technique [0002] Today, as deep learning models become larger and larger, the difficulty of retraining a model in both software and hardware is increasing. Scarce data and expensive computing resources are the problems we face. Due to intellectual property protection and privacy protection, matching training data is often not provided when the deep model is released. [0003] In this case, transfer learning can provide a good solution when we face difficulties such as scarce data and expensive computing resources that are difficult to obtain. Transfer learning uses the s...

AI Technical Summary

Problems solved by technology

[0006] According to the current research status, there are many forms of transfer learning, but no researchers have proposed this cross-dimensional knowledge transfer method before.

Images