BERT model pre-training method, computer device and storage medium

Abstract

The invention discloses a BERT model pre-training method, which comprises the steps such as: loading a BERT model, calling a plurality of attention mechanism modules by the BERT model, obtaining a mask matrix, respectively inputting training data, a knowledge matrix and the mask matrix into each attention mechanism module for processing, and obtaining the output of each attention mechanism module; and performing character string splicing and linearization processing on the output of each attention mechanism module to obtain a semantic vector, determining a training loss value according to a comparison result of the semantic vector and the mask matrix, and adjusting network parameters of the attention mechanism modules. The knowledge matrix can be directly embedded into the multi-head attention mechanism of the BERT model, and the synonym understanding and recognition capability of the BERT model in a text matching task can be enhanced even if a pre-training task using external knowledge is not introduced to pre-train the BERT model. The invention can be widely applied to the technical field of natural languages.

Description

technical field [0001] The invention relates to the technical field of natural language, in particular to a BERT model pre-training method, a computer device and a storage medium. Background technique [0002] The full name of BERT is Bidirectional Encoder Representations from Transformers, which is a deep learning model based on Transformers architecture and encoder. After the BERT model has been pre-trained with unlabeled training data, it only needs to use a small amount of corresponding sample data for specific downstream processing tasks before being applied to specific downstream processing tasks to perform a small amount of training to be able to process downstream processing tasks. , this feature of the BERT model is very suitable for applications in natural language processing (NLP, Natural Language Processing) and other fields. At present, when the BERT model is applied to natural language processing, it still lacks the ability to understand and utilize synonyms. ...

AI Technical Summary

Problems solved by technology

The basic principle of the K-BERT model and the ERNIE model is to introduce additional pre-training tasks using external knowledge in the process of pre-training the BERT model. However, these related technologies will affect the BERT model itself after introducing additional pre-training tasks. The structure and parameters of the BERT model lead to instability in the direction and magnitude of performance changes

Images