Neural network weight initialization method based on transfer learning

Abstract

The invention belongs to the technical field of neural network models, and particularly relates to a neural network weight initialization method based on transfer learning. According to the method, for a specified target task, a neural network model, namely a teacher model, with high complexity is designed, the teacher model is trained, and after training is completed, a generated feature graph isused for guiding weight initialization of a student model; the difference between the feature maps is calculated or the feature maps are mapped into a regeneration kernel Hilbert space, the difference of the feature maps in the regeneration kernel Hilbert space is calculated, and the calculation is simplified by adopting a kernel function method. According to the method, a simple student model isenabled to achieve a better weight initialization effect, and general training is performed on the student model after weight initialization is completed so that the student model is enabled to achieve a better global convergence point and the performance of the student model is enabled to be more excellent. According to the method, the performance of the student model can be effectively improvedon the premise of not increasing the complexity of the student model.

Description

Technical field [0001] The invention belongs to the technical field of neural network models, and specifically relates to a neural network weight initialization method based on knowledge transfer learning. Background technique [0002] Neural networks have made great progress in recent years, especially in the field of computer vision and natural language processing. Many of their performances have surpassed human beings. However, the excessively high computational load and excessive training requirements of neural networks have made neural networks practical Great obstacles were created in the application. Therefore, how to make a lightweight model perform better has become a hot issue that needs to be solved. [0003] In the past few years, many researchers have proposed various solutions to help neural networks achieve a better convergence effect. It mainly includes the following categories. One is based on knowledge distillation and knowledge transfer. It tries to add some ad...

AI Technical Summary

Problems solved by technology

[0002] Neural networks have made great progress in recent years, especially in the fields of computer vision and natural language processing, and many of their performances have surpassed those of humans. posed a major obstacle in the application

Images