Pre-training method and device, equipment and storage medium

Abstract

The invention discloses a pre-training method and device, equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: obtaining an initial text sentence after character masking processing; obtaining a target text sentence based on the initial text sentence after the character masking processing and the additional characters before the sentence; a mask matrix corresponding to the target text sentence is determined, the mask matrix comprises a plurality of elements, each element is used for indicating the operation association degree of two characters corresponding to the element in the target text sentence in the feature extraction process to a to-be-trained feature extraction model, and the element corresponding to the additional character in front of the sentence is not 0; and training a to-be-trained feature extraction model based on the initial text sentence, the target text sentence and the mask matrix. By adopting the method and the device, not only can the feature vector corresponding to each character in the target text sentence be obtained, but also the feature vector corresponding to the target text sentence can be obtained, other training does not need to be carried out additionally, and data operation resources and operation time are reduced.

Description

technical field [0001] The present application relates to the field of computer technology, in particular to a pre-training method, device, equipment and storage medium. Background technique [0002] With the continuous development of machine learning technology, pre-training models are widely emerging. The pre-trained model is a pre-trained model using a large number of unlabeled training sets. After the pre-training is completed, according to different needs, a small number of labeled training sets can be used for targeted training to obtain a trained model. For example, if you want to get a model for predicting the emotion of a text sentence (for example, predicting whether a text sentence is a happy mood or a sad mood, etc.), you can first use a BERT (Bidirectional Encoder Representation from Transformers, based on the two-way encoding of the converter Representation) model is pre-trained to obtain the pre-trained BERT model. Then, add a classifier after the BERT model...

AI Technical Summary

Problems solved by technology

[0005] The embodiment of the present application provides a pre-training method, which can solve the problem in the

Images