A pre-training language model continuous learning method based on general sub-network pruning

Abstract

The application provides a pre-training language model continuous learning method based on general sub-network pruning. The method sets zero to redundant neurons in the network through general sub-network pruning, thereby compressing original knowledge, and then pre-trains and updates the idle neurons set to zero on new professional field data, performs field collaborative continued pre-training, increases the field knowledge contained in the model under the condition that the size of the model remains unchanged, and the different field knowledge can work collaboratively, thereby achieving the effect of knowledge transfer.

Images