BERT model training method and system based on multiplier alternating direction method

Abstract

The invention discloses a BERT model training method and system based on a multiplier alternating direction method. The method comprises the steps that S1, statements to be trained are extracted from a training set, word vectors are extracted, and then the word vectors are input into a BERT model; s2, when the BERT model trains an input word vector, solving a target function by using a multiplier alternating direction method, determining the target function, adding a limiting condition to the input word vector, converting the determined target function into an enhanced Lagrange function, and obtaining an enhanced Lagrange function; solving variable parameters in the target function and an output result of the BERT model by solving the enhanced Lagrange function; s3, variable parameters solved in the objective function are updated until training is completed, and a final BERT model training result is obtained and output. The method has the advantages that the problems of gradient disappearance and explosion in the training process can be avoided, parallel implementation is easy, the training efficiency is high, and the training performance is good.

Description

technical field [0001] The invention relates to the technical field of information extraction, in particular to a BERT model training method and system based on a multiplier alternating direction method. Background technique [0002] With the rapid development of the Internet in recent years, the network media generates massive amounts of unstructured information such as news, pushes, announcements, and notifications every day. For these massive amounts of information, information extraction technology is to quickly and effectively capture valuable information from this information to help analyze some specific information and make decisions based on this specific information. Event extraction is a branch of information extraction technology, which mainly extracts the event information that users are most interested in from unstructured natural language text and displays it in a structured form. Event extraction has a wide range of applications in many fields, such as build...

AI Technical Summary

Problems solved by technology

[0006] 1. Gradient disappearance and gradient explosion: Since the gradient in the backpropagation process has a strong dependence, if the gradient value is too large or too small, it will affect the parameters of the model training, resulting in a decline in the performance of the final model, resulting in gradient disappearance and gradient explosion Problems, especially when applied to application fields such as event extraction (or text classification), because the model needs to be calculated according to the context information of the input sequence, during the reverse calculation process, the calculation of the gradient value of the target loss function is prone to the above problems, and then As a result, BERT model training is difficult, performance is poor, and there will be problems that cannot solve long text encoding

[0007] 2. GPU memory is limited: the number of parameters of the BERT model needs to depend on the matrix multiplication scale in the model, while the number of parameters of the BERT model for event extraction is large, and it needs to consume a large a

Images