Editing method and device of pre-trained language model

Abstract

The embodiment of the application discloses a pre-training language model editing method and device. The main technical scheme comprises: inputting a knowledge sequence to be edited into a pre-training language model; obtaining a first weight representation corresponding to the knowledge sequence from a target network layer included in the pre-training language model, and performing gradient decomposition on the first weight representation to obtain a second weight representation corresponding to each knowledge sample; inputting the second weight representation corresponding to each knowledge sample into an auxiliary network; using the auxiliary network to perform attention processing between each Token in the knowledge sample and attention processing between the knowledge samples for each knowledge sample respectively, and obtaining a fourth weight representation corresponding to the knowledge sequence; updating the first weight representation of the target network layer in the pre-training language model by using the fourth weight representation; and updating the parameters of the auxiliary network by using a loss function obtained by the pre-training language model. Through the application, the pre-training language model can be effectively and efficiently updated with new knowledge.

Images