A sequence labeling acceleration method based on neural network early retirement mechanism

Abstract

The present invention provides a sequence labeling acceleration method based on a neural network early-exit mechanism, which is used to determine whether a pre-trained language model should be early-exited when performing a sequence labeling task. The method comprises the following steps: for a pre-trained language model with L layers, an internal classifier is added after each layer. When performing a sequence labeling task, the sentence-level sequence labeling early-exit mechanism or the word-level sequence labeling early-exit mechanism is used to determine whether the pre-trained language model should be early-exited. The sequence labeling acceleration method based on a neural network early-exit mechanism can be applied to relatively basic sequence labeling tasks in natural language processing, accelerating the model by 3-4 times while only causing a slight performance loss to the model.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and natural language processing, and specifically relates to a sequence labeling acceleration method based on a neural network early retirement mechanism. Background Art

[0002] Sequence labeling is a fundamental task in natural language processing (NLP), including tasks such as word segmentation, part-of-speech tagging, and named entity recognition. It serves as an upstream component of many tasks and is crucial for many downstream tasks. In recent years, the development of pre-trained language models has significantly improved sequence labeling. However, due to their typically large number of parameters and slow inference time, their practical application in real-world scenarios is limited. Pre-trained language models are typically static, performing the same computations on all inputs. Consequently, some computations are redundant for relatively simple examples. Recently, an early-exit mechanism for pre-trained models has been proposed to accelerate inference in pre-trained language models.

[0003] The early exit algorithm consists of two main parts: 1. Training the internal classifier and 2. Customizing the early exit strategy. Internal classifiers are classifiers that are connected after the intermediate layers (not the final layer) of the pre-trained model. With appropriate training strategies, these internal classifiers can provide predictions based on the representations provided by the corresponding layer. The confidence index reflects the reliability of the internal classifier's predictions. If the result is relatively reliable, the prediction is directly output without further calculations. If it is not reliable enough, the input text is fed into the next layer for calculations of deeper representations.

[0004] Currently, most pre-training model acceleration methods based on early retirement mechanisms are designed for text classification tasks, and the confidence calculation in them cannot be directly transferred to sequence labeling scenarios. Summary of the Invention

[0005] The present invention is made to solve the above problems, and its purpose is to provide a sequence labeling acceleration method based on the neural network early retirement mechanism.

[0006] The present invention provides a sequence labeling acceleration method based on a neural network early-retirement mechanism, which is used to determine whether a pre-trained language model is to be retired early when performing a sequence labeling task. The method has the following characteristics: for a pre-trained language model of L layers, an internal classifier is added after each layer. When performing a sequence labeling task, whether the pre-trained language model is to be retired early is determined by a sentence-level sequence labeling early-retirement mechanism or a word-level sequence labeling early-retirement mechanism. The sentence-level sequence labeling early-retirement mechanism includes a sentence-level training stage and a sentence-level reasoning stage. The sentence-level training stage is to jointly train the pre-trained language model of L layers and the added L internal classifiers. The sentence-level reasoning stage is: when the pre-trained language model is executed to the Lth layer, The uncertainty of each word is calculated based on the label probability distribution of each word output by the Lth layer, and then the uncertainty of the entire sentence is obtained by maximum pooling, and the exit is judged based on the early exit uncertainty limit. The word-level sequence labeling early exit mechanism includes the word-level training stage and the word-level reasoning stage. The word-level training stage is to first conduct joint training of the internal classifier and then conduct self-sampling training. The word-level reasoning stage is to calculate the window uncertainty of each word and judge the exit of each word individually based on the minimum exit uncertainty. After each layer of the pre-trained language model, the non-exited words are labeled and classified. If the distribution confidence of the predicted label of the non-exited word is high enough, the predicted label is directly output, and the representation update is stopped and copied directly to the subsequent layer.

[0007] The sequence tagging acceleration method based on the neural network early-retirement mechanism provided by the present invention may also have the following characteristics: wherein the loss function in the sentence-level training phase is:

[0008]

[0009]

[0010] In formula (1), f (l) (X; θ) n is the label probability distribution of the nth word in the Lth layer of the pre-trained language model, H is the cross entropy, and w l is the weight of the loss function of each layer, w l =l.

[0011] The sequence tagging acceleration method based on the neural network early-retirement mechanism provided by the present invention may also have the following characteristics: wherein the uncertainty of each word is as follows:

[0012]

[0013] In formula (3), is the uncertainty of the nth word in the Lth layer, is the label classification probability of the nth word in the Lth layer, C is the total number of label types,

[0014] Then use the maximum pooling method to get the uncertainty of the entire sentence as follows:

[0015]

[0016] The sequence labeling acceleration method based on the neural network early exit mechanism provided by the present invention may also have the following features: wherein, after the joint training of each internal classifier, the self-sampling training is to use the trained pre-trained language model combined with the window size and minimum exit uncertainty obtained by random sampling to obtain the exit layer number of each word corresponding to the input, and then execute the pre-trained language model with a stop update mechanism according to the layer number information to perform model training.

[0017] The sequence tagging acceleration method based on the neural network early-retirement mechanism provided by the present invention may also have the following feature: wherein the window uncertainty is expressed by calculating the maximum uncertainty of all words in the window where each word is located to represent the uncertainty of the word, and the calculation formula is as follows:

[0018]

[0019] In formula (5), k is the window size.

[0020] Functions and effects of the invention

[0021] According to the present invention, a sequence labeling acceleration method based on a neural network early exit mechanism is proposed to use a sentence-level sequence labeling early exit mechanism or a word-level sequence labeling early exit mechanism to determine whether the pre-trained language model should be early exited. For the word-level sequence labeling early exit mechanism, a better exit timing is obtained by calculating the window uncertainty, and self-sampling training is added during the training phase to reduce the difference between training and inference. Compared with previous acceleration methods based on early exit mechanisms, the present invention's sequence labeling acceleration method based on a neural network early exit mechanism can be applied to more basic sequence labeling tasks in natural language processing, and can accelerate by 3-4 times while only causing a slight performance loss to the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flowchart of a sequence labeling acceleration method based on a neural network early retirement mechanism in an embodiment of the present invention;

[0023] Figure 2 1 is a flow chart of a sentence-level sequence annotation early exit mechanism in an embodiment of the present invention;

[0024] Figure 31 is a flow chart of a word-level sequence annotation early exit mechanism in an embodiment of the present invention;

[0025] Figure 4 1 is a performance acceleration ratio curve diagram of the sequence labeling acceleration method based on the neural network early retirement mechanism in an embodiment of the present invention on six data sets. DETAILED DESCRIPTION

[0026] In order to make the technical means and effects achieved by the present invention easier to understand, the present invention is described in detail below with reference to embodiments and drawings.

[0027] <Example>

[0028] Figure 1 This is a flowchart of a sequence labeling acceleration method based on a neural network early retirement mechanism in an embodiment of the present invention.

[0029] like Figure 1 As shown, a sequence labeling acceleration method 100 based on a neural network early retirement mechanism of this embodiment is used to determine whether a pre-trained language model is to be retired early when performing a sequence labeling task, including the following steps:

[0030] For the L-layer pre-trained language model, an internal classifier is added after each layer. When performing sequence labeling tasks, the sentence-level sequence labeling early exit mechanism or the word-level sequence labeling early exit mechanism is used to determine whether the pre-trained language model should be early exited.

[0031] In this embodiment, in a traditional neural network model, the input is generally processed by all L layers of neural networks, and finally the classifier f placed at the top layer L Output the prediction result. According to the sentence-level sequence tagging early exit mechanism or the word-level sequence tagging early exit mechanism of this embodiment, any intermediate classifier can exit.

[0032] The sentence-level sequence tagging early exit mechanism includes the sentence-level training phase and the sentence-level reasoning phase.

[0033] The sentence-level training phase is to jointly train the L-layer pre-trained language model and the added L internal classifiers.

[0034] The loss function of the sentence-level training phase is:

[0035]

[0036]

[0037] In formula (1), f (l) (X; θ) nis the label probability distribution of the nth word in the Lth layer of the pre-trained language model, H is the cross entropy, and w l is the weight of the loss function of each layer, w l =l.

[0038] The sentence-level reasoning stage is as follows: when the pre-trained language model is executed to the Lth layer, the uncertainty of each word is calculated based on the label probability distribution of each word output by the Lth layer, and then the uncertainty of the entire sentence is obtained by maximum pooling, and whether to exit is determined based on the early exit uncertainty limit.

[0039] The uncertainty of each word is as follows:

[0040]

[0041] In formula (3), is the uncertainty of the nth word in the Lth layer, is the label classification probability of the nth word in the Lth layer, C is the total number of label types,

[0042] Then use the maximum pooling method to get the uncertainty of the entire sentence as follows:

[0043]

[0044] Figure 2 1 is a flow chart of a sentence-level sequence annotation early exit mechanism in an embodiment of the present invention.

[0045] like Figure 2 As shown, in this embodiment, taking the three layers in the pre-trained language model as an example, "Dell foundedDell in 1984" is judged whether it is early exited through the sentence-level sequence annotation early exit mechanism. When making the early exit judgment, when the pre-trained language model is executed to the i, i+1, and i+2 layers, the uncertainty of each word is calculated according to the label probability distribution of each word output by the corresponding i, i+1, and i+2 layers, and then the uncertainty of the entire sentence is obtained by maximum pooling. In this embodiment, exit is judged at the i+2 layer according to the early exit uncertainty limit.

[0046] The word-level sequence tagging early-retirement mechanism includes the word-level training phase and the word-level reasoning phase.

[0047] The word-level training phase is to first conduct joint training of the internal classifiers, and then conduct self-sampling training.

[0048] In this embodiment, the loss function when performing joint training of internal classifiers in the word-level training stage is consistent with the loss function in the sentence-level training stage.

[0049] Self-sampling training is to use the trained pre-trained language model in combination with the window size and minimum exit uncertainty obtained by random sampling to obtain the exit layer number of each word corresponding to the input after jointly training each internal classifier. Then, the pre-trained language model with a stop-update mechanism is executed according to the layer number information to perform model training.

[0050] In this embodiment, since the labels of each word in the sequence labeling task are not independent but interdependent, the uncertainty of each word should not be calculated independently. Instead, the uncertainty of the word is represented by calculating the window uncertainty.

[0051] Window uncertainty expresses the uncertainty of a word by calculating the maximum uncertainty of all words in the window where each word is located. The calculation formula is as follows:

[0052]

[0053] In formula (5), k is the window size.

[0054] Figure 3 1 is a flow chart of a word-level sequence tagging early exit mechanism in an embodiment of the present invention.

[0055] like Figure 3 As shown, in this embodiment, taking the three layers in the pre-trained language model as an example, "Dell foundedDell in 1984" is judged whether it exits early through the word-level sequence labeling early exit mechanism. When making the early exit judgment, the window uncertainty of each word is calculated and the exit of each word is judged separately according to the minimum exit uncertainty. Moreover, after each layer of the pre-trained language model, the words that have not exited are labeled and classified. If the distribution confidence of the predicted labels of the words that have not exited is high enough, the predicted labels are directly output, and the representation update is stopped and directly copied to the subsequent layers. When all words have exited, the sequence labeling prediction of the sentence is terminated. In this embodiment, early exit is judged at the i+2th layer.

[0056] In this embodiment, the pre-trained language model BERT is selected, and six data sets including CoNLL2003, Ontonotes4.0, CTB5POS, UD POS, CTB5 word segmentation, and UD word segmentation are used. Performance experiments are conducted in each data set using the sentence-level sequence labeling early exit mechanism (SentEE, full name Sentence-Level Early-Exit) and the word-level sequence labeling early exit mechanism (TokenEE, full name Token-Level Early-Exit) of the sequence labeling acceleration method based on the neural network early exit mechanism of the present invention. Figure 41 is a performance acceleration ratio curve diagram of the sequence labeling acceleration method based on the neural network early retirement mechanism in an embodiment of the present invention on six data sets.

[0057] like Figure 4 As shown in the figure, both SentEE and TokenEE can achieve a certain acceleration effect. As the acceleration ratio gradually increases, the model performance first remains almost unchanged and then slowly decreases. The inflection point of the decline of TokenEE arrives later, and the rate of slow decline is also slower than that of SentEE, which shows that TokenEE has higher acceleration potential.

[0058] Functions and Effects of the Embodiments

[0059] According to the sequence labeling acceleration method based on the neural network early exit mechanism involved in this embodiment, a sentence-level sequence labeling early exit mechanism or a word-level sequence labeling early exit mechanism is proposed to determine whether the pre-trained language model should be early exited, and for the word-level sequence labeling early exit mechanism, a better exit timing is obtained by calculating the window uncertainty, and self-sampling training is added in the word-level training stage to reduce the difference between training and reasoning. Compared with the previous acceleration method based on the early exit mechanism, the sequence labeling acceleration method based on the neural network early exit mechanism of this embodiment can be applied to the more basic sequence labeling tasks in natural language processing, and accelerates by 3-4 times while only causing a slight performance loss to the model.

[0060] The above embodiments are preferred examples of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A sequence labeling acceleration method based on a neural network early exit mechanism, used to determine whether a pre-trained language model should be early exited when performing a sequence labeling task, applied to sequence labeling tasks in natural language processing, characterized by: The following steps are involved: For the pre-trained language model of the L layer, an internal classifier is added after each layer. The pre-trained language model is a static model. The internal classifier can give a prediction result based on the representation given by the corresponding layer. When performing the sequence labeling task, the sentence-level sequence labeling early exit mechanism or the word-level sequence labeling early exit mechanism is used to determine whether the pre-trained language model should be exited early, and any intermediate classifier is exited. The sentence-level sequence tagging early exit mechanism includes a sentence-level training phase and a sentence-level reasoning phase. The sentence level training phase is to jointly train the L-layer pre-trained language model and the added L internal classifiers. Among them, the loss function of the sentence level training stage is: In formula (1), f (l) (X; θ) n is the label probability distribution of the nth word in the Lth layer of the pre-trained language model, H is the cross entropy, and w l is the weight of the loss function of each layer, w l =l; The sentence-level reasoning stage is as follows: when the pre-trained language model is executed to the Lth layer, the uncertainty of each word is calculated based on the label probability distribution of each word output by the Lth layer, and then the uncertainty of the entire sentence is obtained by maximum pooling, and whether to exit is determined based on the early exit uncertainty limit. The uncertainty of each word is as follows: In formula (3), is the uncertainty of the nth word in the Lth layer, is the label classification probability of the nth word in the Lth layer, C is the total number of label types, Then, the uncertainty of the entire sentence is obtained by using the maximum pooling method as follows: The word-level sequence tagging early-retirement mechanism includes a word-level training phase and a word-level reasoning phase. The word-level training phase first performs joint training of the internal classifiers and then performs self-sampling training. The word-level reasoning stage is to calculate the window uncertainty of each word and make a separate judgment on the exit of each word based on the minimum exit uncertainty. The window uncertainty is expressed by calculating the maximum uncertainty of all words in the window where each word is located. The calculation formula is as follows: In formula (5), k is the window size, and each time a layer of the pre-trained language model is passed, the labels of the words that have not exited are classified. If the distribution confidence of the predicted labels of the words that have not exited is high enough, the predicted labels are directly output, and the representation update is stopped and directly copied to the subsequent layers to accelerate the speed of word segmentation, part-of-speech tagging, and named entity recognition.

2. The sequence labeling acceleration method based on the neural network early retirement mechanism according to claim 1 is characterized by: in, The self-sampling training is to use the trained pre-trained language model in combination with the window size obtained by random sampling and the minimum exit uncertainty to obtain the exit layer number of each word corresponding to the input after jointly training each of the internal classifiers, and then execute the pre-trained language model with a stop update mechanism according to the layer number information to perform model training.

Images