Neural language network model training method and device, equipment and medium

Abstract

The embodiment of the invention provides a neural language network model training method and device, equipment and a storage medium, which are used for reducing the labeling amount of training sample data and improving the training efficiency of a language model. The method comprises the following steps: acquiring training sample data; cyclically executing the following steps until the neural language network model obtained by training meets a preset requirement: predicting the unlabeled training sample data by using the neural language network model obtained by previous training, and determining an identification probability for representing that each training sample data is identified; according to a preset selection strategy, on the basis of the recognition probability of each training sample data, selecting a part of training sample data requests from the training sample data which are not labeled for manual labeling; and obtaining the manually labeled training sample data, and training the neural language network model obtained by the previous training based on the manually labeled training sample data to obtain a new neural language network model.

Description

technical field [0001] The present invention relates to the field of neural networks, in particular to a training method, device, equipment and storage medium of a neural language network model. Background technique [0002] In recent years, deep learning methods based on massive data have achieved good results in the text field, but most of the learning process is supervised, that is, a large amount of labeled training data is required. However, the labeling of massive data in real scenarios is not only tedious and time-consuming, but also requires a certain amount of manpower and material resources, such as entity labeling and classification of domain texts. [0003] In order to solve such problems, the concept of transfer learning is proposed, which tries to apply the knowledge obtained from the source task to the target domain. In the field of text, the most common application of transfer learning is the neural language model network, such as ELMo, GPT and BERT models, ...

AI Technical Summary

Problems solved by technology

In the existing technology, the pre-training model obtained by unsupervised learning of the BERT model is first used on a large-scale corpus, and then the transfer learning of a specific text task is carried out. In actual tasks, the training of the language model takes a lot of time, and after the model is trained and identified, it still needs thousands of levels of training data to be labeled, especially for some special application fields, such as medicine, government affairs, etc. Labeling, still need to spend a lot of money

[0004] To sum up, the text deep learning network in the prior art has a large amount of training data labeling, and the training time of the language model is relatively long

Images