A neural network dynamic early regression method based on ensemble learning

Abstract

This invention provides a method for dynamic early termination of neural networks based on ensemble learning, comprising: Step 1, training a multi-layer neural network and its internal classifier using the sum of correlation loss and diversity loss to obtain a trained neural network and a trained internal classifier; Step 2, inputting the data to be processed into the trained neural network, and calculating the intermediate prediction results of the first trained internal classifier for classifying the data in the first layer of the trained neural network to obtain the vote count for each category; Step 3, comparing the maximum number of votes in the intermediate prediction results with a pre-set threshold. If the maximum number of votes is greater than the threshold, the category with the maximum number of votes is taken as the final prediction result, and the network exits from the trained neural network. Otherwise, the network enters the next layer of the trained neural network and votes through the next internal classifier until the maximum value of the new intermediate prediction result is greater than the threshold, at which point the network exits.

Description

Technical Field

[0001] This invention relates to deep learning and natural language processing, and more specifically to a method for dynamic early regression of neural networks based on ensemble learning. Background Technology

[0002] With the development of pre-training techniques, pre-trained language models (such as BERT) have made significant progress in many natural language processing tasks. However, pre-trained language models typically have a large number of parameters and slow inference speed, making them difficult to apply in scenarios with limited computing resources. Most pre-trained language models are static networks, performing the same amount of computation for any input, thus consuming unnecessary computing resources for relatively simple text inputs. To address this, researchers have recently proposed a dynamic early termination mechanism, which allows the neural network to perform only the computation of a portion of its layers and exit the process when a confident result is obtained, without executing subsequent neural network layers.

[0003] The dynamic early termination algorithm mainly consists of two parts: (1) training the internal classifier; and (2) defining the early termination strategy. The internal classifier refers to a classifier inserted into certain intermediate layers of the neural network. After training, these internal classifiers can provide intermediate prediction results based on information from previous layers. By defining a confidence index, the reliability of the intermediate results of the internal classifier can be measured. If the intermediate result is relatively reliable, it can be used as the final prediction result and the algorithm can exit the current layer without performing subsequent calculations. In recent years, the dynamic early termination algorithm has been successfully applied to pre-trained language models, significantly improving the model's inference speed while maintaining relatively high accuracy.

[0004] However, most current dynamic early termination mechanisms only utilize the intermediate results of a single-layer classifier when training the internal classifier and designing the early termination strategy, failing to fully leverage the results of the internal classifiers in previous layers, thus sacrificing some classification accuracy. Therefore, a method is needed to optimize and solve the above problems. Summary of the Invention

[0005] This invention is made to solve the above-mentioned problems, and aims to provide a method for dynamic early regression of neural networks based on ensemble learning.

[0006] This invention provides a method for dynamic early termination of neural networks based on ensemble learning, characterized by the following steps: Step 1, training a multi-layer neural network and its internal classifiers inserted between every two layers using the sum of correlation loss and diversity loss, resulting in a trained neural network and trained internal classifiers; Step 2, inputting the data to be processed into the trained neural network, calculating the intermediate prediction results of the first trained internal classifier for data classification in the first layer of the trained neural network, obtaining the vote count for each category; Step 3, comparing the maximum number of votes in the intermediate prediction results with a pre-set threshold. When the maximum number of votes is greater than the threshold, the category containing the maximum number of votes is taken as the final prediction result, and the network exits the trained neural network. When the maximum number of votes is less than or equal to the threshold, the network enters the next layer of the trained neural network, and the next trained internal classifier performs voting calculations until the maximum value of the new intermediate prediction results is greater than the threshold, at which point the network exits the trained neural network.

[0007] The neural network dynamic early regression method based on ensemble learning provided by this invention may also have the following feature: wherein, in step 1, the formula for the sum L of the correlation loss and the diversity loss is:

[0008]

[0009] In equation (1), L rel L represents the correlation loss function. div Let x represent the diversity loss function. i Let w be the predicted distribution of the i-th internal classifier, y be the correct label, and w be the predicted distribution of the i-th internal classifier. i Let L be the weight of the i-th internal classifier, and there are L internal classifiers in total. Let λ be a hyperparameter L that adjusts the two-term loss function. CE This represents the cross-entropy loss function, which is... x j To match the predicted distribution x of the current internal classifier i The closest internal classifier.

[0010] The neural network dynamic early termination method based on ensemble learning provided by this invention may also have the following feature: wherein, in step 2, the formula for calculating the number of votes is as follows:

[0011]

[0012] In equation (2), c represents the category index, and y c Represents the c-th category label, Pred(x) j ) represents the predicted class of the j-th internal classifier, l k represents the scaling scalar, and k represents the pre-defined hyperparameter.

[0013] The role and effect of invention

[0014] According to the ensemble learning-based neural network dynamic early termination method of the present invention, the internal classifiers are trained by ensemble learning, so that they not only optimize the classification accuracy, but also increase the prediction diversity among them; in addition, based on the trained internal classifiers, the final result is predicted according to the intermediate results of all currently available layers and it is decided whether to terminate early.

[0015] Therefore, the neural network dynamic early termination method based on ensemble learning of the present invention fully develops and utilizes the results of multiple internal classifiers, which not only improves the model inference efficiency, but also improves and maintains the model accuracy. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall framework of the neural network dynamic early termination method based on ensemble learning in an embodiment of the present invention;

[0017] Figure 2 This is a comparison of the speed-accuracy curves of the loss function used in the neural network dynamic early termination method based on ensemble learning in the embodiments of the present invention and the loss function used only by the correlation loss function, i.e., the loss function adopted by most existing methods.

[0018] Figure 3 This is a speed-accuracy curve comparison chart between the neural network dynamic early termination method based on ensemble learning in the embodiments of the present invention and the previous existing method PABEE. Detailed Implementation

[0019] To make the technical means and effects of the present invention easy to understand, the present invention will be specifically described below in conjunction with embodiments and accompanying drawings.

[0020] This invention provides a method for dynamic early regression of neural networks based on ensemble learning, comprising the following steps:

[0021] Step 1: Train a multi-layer neural network and its internal classifier inserted between every two layers of the neural network using the sum of correlation loss and diversity loss, to obtain a trained neural network and a trained internal classifier.

[0022] In this invention, the formula for the sum L of correlation loss and diversity loss is:

[0023]

[0024] In equation (1), L rel L represents the correlation loss function. div Let x represent the diversity loss function. i Let w be the predicted distribution of the i-th internal classifier, y be the correct label, and w be the predicted distribution of the i-th internal classifier.i Let λ be the weight of the i-th internal classifier, and there are L internal classifiers in total. Let λ be a hyperparameter that adjusts the two-term loss function. CE This represents the cross-entropy loss function, which is... x j To match the predicted distribution x of the current internal classifier i The closest internal classifier.

[0025] Furthermore, in practical implementation, the weights of the internal classifiers and the two loss functions of each layer can be finely adjusted according to the model and the task, as shown in the following formula:

[0026]

[0027] Where, α i and β i represents the weights of the correlation loss and diversity loss of the internal classifier in layer i.

[0028] Furthermore, after training with the aforementioned relevance loss function, each internal classifier in the neural network possesses a certain ability to predict the correct labels in advance. Simultaneously, the diversity loss function ensures that the predictions of these internal classifiers exhibit some differences and complementarity, making their predictions easier to integrate for inference.

[0029] Furthermore, the correlation loss, designed to optimize the maximum likelihood between the internal classifier and the correct label, is a loss function used in many dynamic early termination algorithms. The diversity loss function is used to increase the diversity among the prediction distributions of multiple internal classifiers, thereby making them complementary in decision-making. Specifically, the diversity loss makes the prediction distribution x of the current internal classifier more compatible with the prediction distribution of the current internal classifier. i The closest internal classifier x j Maximizing the cross-entropy, since optimizing the cross-entropy is equivalent to optimizing the Kullback-Leibler divergence, this loss function actually widens the distance between the predicted distributions of the two most similar internal classifiers.

[0030] Step 2: Input the data to be processed into the trained neural network. In the first layer of the trained neural network, calculate the intermediate prediction results of the first trained internal classifier for classifying the data, and obtain the number of votes for each category.

[0031] In this invention, the formula for calculating the number of votes is as follows:

[0032]

[0033] In equation (2), c represents the category index, and y c Represents the c-th category label, Pred(x) j ) represents the predicted class of the j-th internal classifier, lk represents the scaling scalar, and k represents the pre-defined hyperparameter.

[0034] Step 3: Compare the maximum number of votes in the intermediate prediction results with a pre-set threshold. If the maximum number of votes is greater than the threshold, the category containing the maximum number of votes is taken as the final prediction result, and the system exits the trained neural network. If the maximum number of votes is less than or equal to the threshold, the system enters the next layer of the trained neural network and performs voting calculations through the next trained internal classifier until the maximum value of the new intermediate prediction result is greater than the threshold, at which point the system exits the trained neural network.

[0035] Example:

[0036] Figure 1 This is a schematic diagram of the overall framework of the neural network dynamic early termination method based on ensemble learning in an embodiment of the present invention.

[0037] In this embodiment, as Figure 1 As shown, the neural network has 4 layers, and the neural network and internal classifiers have been trained. When the inference process reaches the 3rd layer of the neural network, the dynamic early termination mechanism infers the correct label Y by voting according to the results X1, X2, and X3 of the three internal classifiers f1, f2, and f3 using a voting function. The specific process Φ is as follows:

[0038] Step a, in the l-th layer of the neural network, calculate the number of votes for each category for the first l internal classifiers:

[0039]

[0040] Here, k is a pre-defined hyperparameter that can take values ​​between (0, 1) during implementation. V l This represents the maximum number of votes obtained by the first l internal classifiers for all category labels.

[0041] Step b, when V l If the value is greater than δ, the neural network exits and the category with the most votes is used as the final predicted category, where δ is a pre-set threshold; otherwise, it enters the next layer of the neural network to continue execution.

[0042] Furthermore, the speedup ratio of the model can be controlled by adjusting the threshold δ. The smaller δ is, the earlier the model exits the layer, and the greater the speedup ratio.

[0043] Figure 2 This is a comparison of the speed-accuracy curves between the loss function used in the neural network dynamic early termination method based on ensemble learning in the embodiments of the present invention and the loss function used only by the correlation loss function, which is the loss function adopted by most existing methods. Figure 2(a) A comparison of the speed-accuracy curves of the ALBERT model using the loss function of this embodiment versus using only the correlation loss function on the MRPC dataset. Figure 2 (b) A comparison of the speed-accuracy curves of the ALBERT model using the loss function of this embodiment versus using only the correlation loss function on the SST-2 dataset. Figure 2 (c) A comparison of the speed-accuracy curves of the ALBERT model using the loss function of this embodiment versus using only the correlation loss function on the TREC dataset. Figure 2 (d) A comparison of the speed-accuracy curves of the ALBERT model using the loss function of this embodiment versus using only the correlation loss function on the SST-5 dataset. Figure 2 (e) A comparison of the speed-accuracy curves of the BERT model using the loss function of this embodiment versus using only the correlation loss function on the MRPC dataset. Figure 2 (f) A comparison of the speed-accuracy curves of the BERT model using the loss function of this embodiment versus using only the correlation loss function on the SST-2 dataset. Figure 2 (g) A comparison of the speed-accuracy curves of the BERT model using the loss function of this embodiment versus using only the correlation loss function on the TREC dataset. Figure 2 (h) is a comparison of the speed-accuracy curves of the BERT model using the loss function of this embodiment versus using only the correlation loss function on the SST-5 dataset.

[0044] Figure 3 This is a comparison of the speed-accuracy curves between the neural network dynamic early termination method based on ensemble learning in an embodiment of the present invention and the existing method PABEE. Figure 3 (a) A comparison of the speed-accuracy curves of the ALBERT model using the voting-based dynamic early termination strategy of this invention with the previous state-of-the-art methods on the MRPC dataset. Figure 3 (b) A comparison of the speed-accuracy curves of the ALBERT model using the voting-based dynamic early termination strategy of this invention with the previous state-of-the-art methods on the SST-2 dataset. Figure 3 (c) A comparison of the speed-accuracy curves of the ALBERT model using the voting-based dynamic early termination strategy of this invention with the previous state-of-the-art methods on the TREC dataset. Figure 3 (d) A comparison of the speed-accuracy curves of the ALBERT model using the voting-based dynamic early termination strategy of this invention with the previous state-of-the-art methods on the SST-5 dataset. Figure 3 (e) A comparison of the speed-accuracy curves of the BERT model using the voting-based dynamic early termination strategy of this invention with the previous state-of-the-art methods on the MRPC dataset. Figure 3(f) A comparison of the speed-accuracy curves of the BERT model using the voting-based dynamic early termination strategy of this invention with the previous best method on the SST-5 dataset.

[0045] like Figure 2 As shown, with a speedup of 1.3x, ALBERT achieves a prediction accuracy of 92.5% on the MRPC dataset using the loss function of this invention, which significantly exceeds the accuracy of 90.5% achieved using only the correlation loss function.

[0046] like Figure 3 As shown, using a voting-based dynamic early termination method, ALBERT can achieve an accuracy of over 93% on the MRPC dataset while maintaining a speedup of 1.2x, while the previous best method, PABEE, could only achieve a maximum accuracy of 92.5% on the same dataset.

[0047] Therefore, even compared to the previous best method PABEE, this embodiment achieves better accuracy and inference speed on most datasets.

[0048] Furthermore, the neural network in the attached diagram can be any multi-layer encoder, such as BERT, ALBERT, RoBERTa, etc., and the internal classifier can be implemented as a multi-layer perceptron or a linear classifier.

[0049] The role and effect of the embodiments

[0050] As can be seen from the above embodiments, when the neural network dynamic early regression method based on ensemble learning in this embodiment is adopted, the results of the two widely used pre-trained language models ALBERT and BERT on binary and multi-class text classification datasets show that the method of this embodiment can usually achieve higher accuracy at the same speedup ratio.

[0051] According to the ensemble learning-based neural network dynamic early termination method involved in this embodiment, the internal classifiers are trained by ensemble learning, so that they not only optimize the classification accuracy, but also increase the prediction diversity among them; in addition, based on the trained internal classifiers, the final result is predicted according to the intermediate results of all currently available layers and it is decided whether to terminate early.

[0052] Therefore, the neural network dynamic early termination method based on ensemble learning in this embodiment fully develops and utilizes the results of multiple internal classifiers, which not only improves the model inference efficiency, but also improves and maintains the model's accuracy.

[0053] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. An ensemble learning-based neural network dynamic early exit method for improving the accuracy and inference efficiency of a neural network model, the neural network model being a pre-trained language model for text classification, characterized in that, The method comprises the following steps: Step 1, training a multi-layer neural network and an internal classifier inserted between each two layers of the neural network by using a sum of correlation loss and diversity loss, to obtain a trained neural network and a trained internal classifier, wherein the internal classifier is a multi-layer perceptron or a linear classifier; Step 2, inputting data to be processed into the trained neural network, and calculating an intermediate prediction result of the data classified by a first trained internal classifier at a first layer of the trained neural network to obtain a vote number of each category; Step 3, comparing a maximum vote number in the intermediate prediction result with a preset threshold value, when the maximum vote number is greater than the threshold value, taking a category where the maximum vote number is located as a final prediction result, and exiting from the trained neural network, and when the maximum vote number is less than or equal to the threshold value, entering a next layer of the trained neural network, and calculating a vote number by using a next trained internal classifier until a maximum value of a new intermediate prediction result is greater than the threshold value, and then exiting from the trained neural network, Wherein, in the step 1, the sum of the correlation loss and the diversity loss The formula is: （1） In formula (1), represents a correlation loss function, represents a diversity loss function, x i is the prediction distribution of the i-th internal classifier, y is the correct label, w i is the weight of the i-th internal classifier, there are L internal classifiers in total, and λ is a hyperparameter for adjusting the two loss functions, represents a cross-entropy loss function, which is , x j is the internal classifier closest to the prediction distribution x i of the current internal classifier, In step 2, a calculation formula of the vote number is as follows: （2） In formula (2), c represents a category index, y c represents a cth category label, Pred(x j ) represents a predicted category of a jth internal classifier, l k represents a scaling scalar, and k represents a preset hyperparameter.

Images