A neural network binarization quantization method and quantization device for a BERT model

Abstract

The application discloses a neural network binarization quantization method and a quantization device for BERT model, which are used for executing a natural language processing task. In the binarization stage of the BERT model, input data is first subjected to a binarization embedding layer and then is sent to a transformer module; the transformer module comprises a multi-head attention module and a feedforward network; a bidirectional attention mechanism based on information theory is introduced in the multi-head attention module, a binarization representation with maximum information entropy is adopted, so that the binarization model restores the perception of the input data; when the optimization direction of the BERT model is determined, a matching optimization direction is provided by reselecting the extraction parameters, and then a similarity matrix is constructed to eliminate the scale difference and numerical instability. By using the application, the actual demand that both the full precision of the BERT model is retained and the storage consumption and the calculation amount are reduced in the natural language processing task can be met.

Description

Technical Field

[0001] This invention relates to a neural network binarization quantization method for BERT models, and also to a quantization device that uses this neural network binarization quantization method to meet the needs of sentiment analysis scenarios in natural language processing tasks, belonging to the field of natural language processing technology. Background Technology

[0002] In 2018, Google released the BERT (Bidirectional Encoder Representation from Transformers) neural network model, which demonstrated astonishing performance in machine reading comprehension: surpassing human capabilities across all metrics and achieving state-of-the-art results in 11 different NLP (Natural Language Processing) tests at the time. The BERT model brought a landmark change to NLP and is one of the most important recent advancements in the field.

[0003] However, the BERT model requires a huge number of floating-point operations during the training and inference processes of natural language processing, placing high demands on both computing and storage resources, making it difficult to promote its application on edge devices. The full binarization method for neural network models (i.e., 1-bit weights, embeddings, and activations) quantizes the original 32-bit floating-point parameters in the network to a 1-bit representation, saving parameter storage space and minimizing the storage consumption of the neural network model. Simultaneously, it transforms the calculation of parameters from floating-point operations to bitwise operations, reducing the computational load of the neural network and greatly accelerating the network inference process.

[0004] Unfortunately, full binarization of the BERT model typically results in a significant performance degradation, and the model even crashes in natural language processing tasks. Previous research has pushed weights and embeddings to binarization, but none has been able to accurately binarize a BERT model with 1-bit activation. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the primary technical problem to be solved by the present invention is to provide a neural network binarization and quantization method for BERT models.

[0006] Another technical problem to be solved by the present invention is to provide a quantization device (also known as a natural language processing device) that adopts the neural network binarization quantization method to meet the needs of sentiment analysis scenarios in natural language processing tasks.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] According to a first aspect of the present invention, a neural network binarization and quantization method for BERT models is provided for performing natural language processing tasks, comprising the following steps:

[0009] In the binarization stage of the BERT model, the input data first passes through the binarization embedding layer and is then fed into the converter module; the converter module includes a multi-head attention module and a feedforward network.

[0010] A bidirectional attention mechanism based on information theory is introduced into the multi-head attention module, and a binary representation with maximized information entropy is adopted to enable the binary model to restore its perception of the input data.

[0011] When determining the optimization direction of the BERT model, the first step is to provide a matching optimization direction by reselecting the extracted parameters, and then a similarity matrix is ​​constructed to eliminate scale differences and numerical instability.

[0012] Preferably, in the bidirectional attention mechanism, the attention score is binarized using a bool function, wherein the element with the lower value in the attention weight is binarized to 0.

[0013] Preferably, the bidirectional attention mechanism is implemented using the following formula:

[0014]

[0015] Among them, B A It is the attention weight of binarization, B V It is the binary value sign(V). It is a bitwise-Affine matrix multiplier, by Composed of displacement.

[0016] Preferably, the extracted parameters are query Q, key K, and value V; wherein...

[0017] Q = bi-linear Q (H), K = bi-linear K (H), V = bi-linear V (H);

[0018] bi-linear Q bi-linear K bi-linear V These represent three different binary linear layers.

[0019] Preferably, in the binarized linear layer, a pre-binarization weight with zero mean is applied to maximize the binarization weight and activation information.

[0020] Among them, the attention score A is preferred. Tl The output M of the multi-head attention module Tl and hidden state H Tl The data is extracted from the full-precision teacher network in a hierarchical manner and then transferred to the binarized student network.

[0021] Preferably, mean squared error is used as the loss function to measure the difference between the student network and the teacher network on the corresponding features, and then the prediction layer extraction loss is obtained by minimizing the soft cross-entropy between the teacher's logical value and the student's logical value.

[0022] According to a second aspect of the present invention, a neural network binarization and quantization apparatus for BERT models is provided for performing natural language processing tasks. The apparatus includes a processor and a memory, wherein the processor reads a computer program from the memory to perform the following operations:

[0023] In the binarization stage of the BERT model, the input data first passes through the binarization embedding layer and is then fed into the converter module; the converter module includes a multi-head attention module and a feedforward network.

[0024] A bidirectional attention mechanism based on information theory is introduced into the multi-head attention module, and a binary representation with maximized information entropy is adopted to enable the binary model to restore its perception of the input data.

[0025] When determining the optimization direction of the BERT model, the first step is to provide a matching optimization direction by reselecting the extracted parameters, and then a similarity matrix is ​​constructed to eliminate scale differences and numerical instability.

[0026] Preferably, the input data consists of sentence number and sentence content from a natural language processing task, and the corresponding output data is the sentence sentiment type.

[0027] Compared with existing technologies, the neural network binarization and quantization method for BERT models provided in this invention introduces an efficient Bi-Attention mechanism to statistically maximize representation information, and utilizes Direction Matching Extraction (DMD) to accurately optimize the binarized BERT model. Using this invention, the practical needs of natural language processing tasks can be met by preserving the full accuracy of the BERT model while reducing storage consumption and computational load. Attached Figure Description

[0028] Figure 1 This is a schematic diagram illustrating the data processing process of the neural network binarization and quantization method for the BERT model provided in this invention.

[0029] Figure 2A flowchart of the neural network binarization and quantization method for the BERT model provided by this invention;

[0030] Figure 3 This is a comparison diagram of the implementation effects of the direction matching extraction method in an embodiment of the present invention;

[0031] Figure 4 This is an example diagram of a natural language processing device employing the neural network binarization and quantization method provided by the present invention. Detailed Implementation

[0032] The technical content of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] Through in-depth research, the inventors discovered that the performance of the BERT model during binarization stems from information degradation in the bidirectional attention mechanism during forward propagation and a mismatch in the optimization direction extracted during backward propagation. First, the bidirectional attention mechanism allows the BERT model to selectively pay attention to parts of the input while ignoring irrelevant content. However, the inventors' analysis shows that direct binarization leads to almost complete information degradation of the attention weights, rendering the selective ability of the bidirectional attention mechanism ineffective. Second, the extraction of information in a fully binarized BERT model utilizes attention scores, which are the direct combination of the two binarized activations. However, the inventors demonstrate that this leads to a severe mismatch in the optimization direction, as there is a non-negligible error between the default and expected optimization directions.

[0034] To address this, this invention first provides a neural network binarization and quantization method for the BERT model, meeting the practical needs of natural language processing tasks that preserve the full accuracy of the BERT model while reducing storage consumption and computational load. Specifically, to solve the information degradation problem of the bidirectional attention mechanism, an efficient Bi-Attention mechanism is introduced based on information theory. The Bi-Attention mechanism applies a binarized representation that maximizes information entropy, enabling the binarized BERT model to recover its perception of the input data. Furthermore, this invention proposes a Direction Matching Extraction (DMD) method to eliminate direction mismatches during extraction. The DMD method employs appropriate activation and utilizes the knowledge of the similarity matrix constructed during extraction for precise optimization. The combination of these two improvements allows for the construction of a fully binarized BERT model (referred to as the BiBERT model).

[0035] The following is a detailed explanation of this.

[0036] First, the inventors will give a brief introduction to the basic structure of the binarized BERT model. Generally speaking, the forward and backward propagation of the sign function in a binarized network can be expressed as:

[0037]

[0038]

[0039] Here, C is the cost function for mini-batch operations. The sign function is used in forward propagation, while the straight-through estimator (STE) is used in backward propagation to obtain the derivative.

[0040] Regarding the weights of the binarized linear layer, in this embodiment of the invention, the weights are redistributed to zero mean to preserve representational information, and a scaling factor is applied to minimize quantization error. To improve computational efficiency, the activations are symbolically binarized without rescaling. Therefore, the calculation result can be expressed as:

[0041]

[0042] Where W and X represent the full-precision weights and activations, μ(·) represents the mean, and α w It is the scaling factor for the weights. This represents matrix multiplication.

[0043] like Figure 1 As shown, the input data first passes through a binarized embedding layer and then is fed into the transformer block. Each transformer block consists of two parts: a multi-head attention (MHA) module and a feed-forward network (FFN). The computation of the MHA module depends on the query Q, the key K, and the value V, all of which are derived from the hidden states. Where N represents the sequence length and D represents the feature dimension. For a specific transformation layer, the computation in the attention head can be represented as:

[0044] Q = bi-linear Q (H), K = bi-linear K (H), V = bi-linear V (H) (2)

[0045] Where, bi-linear Q bi-linear K bi-linear V These represent three different binary linear layers. The inventors then calculated the attention score A as follows:

[0046]

[0047] Among them, B Q and B K These are the binarized query and key, respectively. It's important to note that the resulting attention weights are truncated by an attention mask, and each row in A can be viewed as a k-dim vector, where k is the number of unmasked elements. The inventors then binarized the attention weights as follows:

[0048]

[0049] The inventors followed the original BERT model architecture, incorporating the MHA module and the remainder of the feedforward network into a binarized network.

[0050] Next, we will introduce the distillation step for the binarized BERT model in this embodiment of the invention. The distillation step can alleviate the performance degradation of the quantized BERT model under ultra-low bit-width settings, and it can be applied seamlessly to any architecture. One feasible approach is to use the attention score A... Tl The output of the MHA module is M Tl and hidden state H Tl The data is extracted hierarchically from the full-precision teacher network and then transferred to the binarized student counterparts network. In one embodiment of the invention, the inventors use mean squared error (MSE) as the loss function to measure the differences between the student and teacher networks on corresponding features.

[0051]

[0052] Then, by minimizing the teacher's logical value (y) T The soft cross-entropy (SCE) between the student's logistic value (y) and the student's logistic value (y) is used to calculate the loss for the prediction layer. The objective function is expressed as:

[0053]

[0054] The inventors' research shows that the BERT model suffers significant information degradation during forward propagation due to the bidirectional attention mechanism and severe optimization direction mismatch during backward propagation. To address the information degradation problem of binary representations in forward propagation, the inventors propose an information theory-based Bi-Attention mechanism. This mechanism statistically maximizes the entropy of the representation and restores the bidirectional attention mechanism in the fully binary BERT model. A detailed explanation follows.

[0055] Because the ability to compress bit-width representations (weights, activations, and embeddings) in a fully binarized BERT model is limited, an ideal binarized representation should preserve the given full-precision counterpart as much as possible. This means that the mutual information between the binarized representation and the full-precision representation should be maximized. When a deterministic sign function is applied to the binarization of a BERT model, the objective is equivalent to maximizing the information entropy of the binarized representation. The information entropy B is maximized, defined as:

[0056]

[0057] Here, B∈{-1,1} is a random variable sampled from the information entropy B by the probability mass function p. Therefore, the information entropy of the binarized representation should be maximized to better preserve the full-precision correspondence and allow the bidirectional attention mechanism to function well. Applying zero-mean pre-binarization weights in the binarized linear layer is an effective practice, as it maximizes the information of both the binarization weights and the activations.

[0058] For the attention structure in the full-precision BERT model, the normalized attention weights obtained through the softmax function are crucial. However, directly applying the binarization function leads to a complete loss of information in the binarized attention weights. Specifically, since softmax(A) is considered to follow a probability distribution, its elements... All elements are quantized to 1, while the information entropy degenerates to 0. A common measure to mitigate this information degradation is to shift the distribution of the input tensor before applying the sign function, which can be expressed as:

[0059]

[0060] The shift parameter τ is also considered as the threshold for binarization, with the aim of maximizing the entropy after binarization. Furthermore, It is fixed in the reasoning process.

[0061] To mitigate the information degradation caused by binarization in the bi-attention mechanism, the inventors introduced an efficient Bi-Attention mechanism into the fully binarized BERT model. This mechanism statistically maximizes the information entropy of the binarized representation and applies bitwise operations for fast inference. A detailed explanation follows.

[0062] First, the inventors noted that the softmax function is order-preserving, which means that there exists a fixed threshold φ(τ,A) that maximizes the information entropy (A-φ(τ,A) of the binary representation.

[0063] To restore the bidirectional attention mechanism for capturing key factors, the inventors binarized the attention weights into Boolean values, and their design is driven by maximizing information entropy. In the Bi-Attention mechanism, the inventors used a boolean function to binarize the attention score A, defined as:

[0064]

[0065] By applying the boolean function, elements with lower values ​​in the attention weights are binarized to 0. Therefore, the attention weight with the highest entropy value can filter out elements in the critical part. The provided bidirectional attention mechanism is ultimately represented as:

[0066]

[0067]

[0068] Among them, B V It is the binary value sign(V), B A These are the attention weights in binarization. It is a meticulously designed Bitwise-Affine Matrix Multiplication (BAMM) arithmetic unit, by Composed of shifts, it is used to align training and inference representations and perform efficient bit computation.

[0069] In summary, in the Bi-Attention mechanism provided in this embodiment of the invention, the information entropy of the binarized attention weights is maximized to alleviate the significant information degradation caused by binarization and restore the bi-attention mechanism. By excluding the softmax function, this Bi-Attention mechanism also achieves higher efficiency.

[0070] To address the direction mismatch issue that occurs in the backpropagation of the fully binarized BERT model, the inventors further proposed a Direction Matching Extraction (DMD) method. This method effectively utilizes the knowledge of the teacher network by using appropriate extraction selection and a carefully constructed similarity matrix, thereby optimizing the fully binarized BERT model more accurately.

[0071] Formula (5) above shows that the extraction of attention score A in a specific layer can be represented as MSE(A,A) T ), where A and A T These are the attention scores in the binarized BERT model and the fully accurate BERT model, respectively. The attention score in the binarized BERT model is calculated by binarizing the query B... Q and key B K The result of multiplication is the loss. It can be represented as:

[0072]

[0073] Since the attention score is determined by two binarized activations (binarized query B) Q and key B K It is obtained by direct multiplication. Its extraction can also be misled by direction mismatch. See also Figure 3 As shown in the upper part (a), directional mismatch in attention score extraction is common and severe, even leading to a higher degree of optimization in the mismatch direction.

[0074] To address the optimization direction mismatch problem in the binarization extraction of the BERT model, the inventors proposed a redesigned Direction Matching Extraction (DMD) method in the BiBERT model, as detailed below:

[0075] First, instead of attention scores, the upstream query Q and key K are extracted to leverage their knowledge while mitigating orientation mismatch. Furthermore, the inventors also extracted the value V to further cover all inputs to the MHA module.

[0076] Then, the inventors constructed a similarity matrix for extraction and activation, which can be represented as follows:

[0077]

[0078] Here, ||·|| represents L2 normalization. Previous research has shown that matrices constructed in this way are considered to reflect specific patterns in network semantic understanding. The inventors further discovered that, because the matrix focuses more on endogenous relative relationships, it is also numerically scale-normalized and stable, making it suitable for extraction between binary networks and full-precision networks. The extraction loss can be expressed as:

[0079]

[0080] Where L represents the number of layers in the converter. Loss Items and As shown in Equation (6), it is used to supervise the hidden state and the prediction result, respectively.

[0081] The Direction Matching Extraction (DMD) method in this embodiment of the invention first provides an optimized direction for matching by reselecting appropriate extraction parameters, and then constructs a similarity matrix to eliminate scale differences and numerical instability, thereby achieving a fully binarized BERT model through precise optimization. Figure 3 The lower half (b) shows the beneficial effects of this Direction Matching Extraction (DMD) method. From Figure 3As can be seen, although a small number of direction mismatches exist in attention score extraction, their proportion has been greatly reduced.

[0082] In sentiment analysis within natural language processing (NLP) tasks, the BiBERT model described above can be fine-tuned for specific downstream tasks before being applied to that NLP task. When training on a task-specific dataset, the output type and evaluation metric of the BiBERT model need to be set according to the downstream task. For example, for the single-sentence classification task of SST-2, the dataset contains sentences from movie reviews and their sentiment annotations by humans. This task assigns the sentiment of a given sentence, categorized into two classes: positive sentiment (labeled 1) and negative sentiment (labeled 0), using only sentence-level labels. Furthermore, this NLP task is also a binary classification task, at the sentence level, categorized into positive and negative sentiment, with accuracy as the evaluation metric.

[0083] In different embodiments of the present invention, the BiBERT model can be applied to downstream tasks of natural language processing such as sequence classification, label classification, question answering systems, sequence prediction, and sentiment analysis. Taking the single-sentence binary classification task of SST-2 as an example, the input data includes sentence number and sentence content, and the corresponding output data is the sentence sentiment type. The sentiment classification sequence data is divided into five categories according to sentiment values ​​[0,0.2], (0.2,0.4], (0.4,0.6], (0.6,0.8], (0.8,1.0], which correspond to five sentiment categories: "very negative", "negative", "neutral", "positive", and "very positive". In SST-2, "neutral" comments are deleted, "very positive" and "positive" comments are labeled as "positive", and "very negative" and "negative" comments are labeled as "negative". For example, for the input movie review "It's a charming and often affecting journey", the BiBERT model, after pre-training, assigns a sentiment score to the input sequence during fine-tuning and performs sentiment classification based on the sentiment score.

[0084] Based on the aforementioned neural network binarization quantization method for the BERT model, this invention further provides a quantization device (also known as a natural language processing device) that adopts this neural network binarization quantization method to meet the needs of sentiment analysis scenarios in natural language processing tasks.

[0085] like Figure 4As shown, the natural language processing device includes a memory and a processor, and may further include communication components, sensor components, power supply components, multimedia components, and input / output interfaces as needed. The memory, communication components, sensor components, power supply components, multimedia components, and input / output interfaces are all connected to the processor. The memory can be static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc. The processor can be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP) chip, etc. Other communication components, sensor components, power supply components, multimedia components, etc., can be implemented using common components found in existing electronic devices such as personal computers and smartphones, and will not be specifically described here.

[0086] In the above-mentioned natural language processing device, the memory stores the program or instructions of the above-mentioned neural network binarization and quantization method; the processor is coupled to the memory and is used to execute the program or instructions in the memory so that the electronic device executes the neural network binarization and quantization method in the above embodiment to perform downstream tasks of natural language processing, including sequence classification, label classification, question answering system, sequence prediction, sentiment analysis, etc.

[0087] To verify the effectiveness of the BiBERT model in efficiently handling different natural language processing tasks on the GLUE benchmark, the inventors conducted multiple experiments. Details are as follows:

[0088] As shown in Table 1, the fully binarized BERT model suffers significant performance degradation on SST-2, MRPC, RTE, and QQP tasks. Bi-Attention and Direction Matching Extraction (DMD) mechanisms improve performance when used individually, and their combination further enhances performance. In summary, these two techniques can mutually reinforce each other, improving the BiBERT model and further narrowing the performance gap between the fully binarized BERT model and its full-precision counterpart.

[0089] Table 1

[0090]

[0091] As shown in Tables 2 and 3, the inventors compared BiBERT with the SOTABERT quantization method under ultra-low bit width in terms of accuracy and efficiency to fully demonstrate the technical features of the present invention.

[0092] Table 2

[0093]

[0094] Table 3

[0095]

[0096] Table 2 shows the GLUE benchmark results without data augmentation. The inventors' BiBERT model significantly outperforms existing state-of-the-art methods in average accuracy at ultra-low bit widths. Furthermore, the BiBERT model is the first binarization method to achieve convergence on several natural language processing tasks, such as CoLA, MRPC, and RTE, at ultra-low bit widths. With increasing data volume, the potential of the BiBERT model is further unleashed; its powerful binarization representation and accurate optimization even achieve performance comparable to the full-precision BERT model on some tasks, for example, an accuracy of 90.9% on SST-2 (a decrease of only 2.2%) (Table 3). These results demonstrate that the BiBERT model fully utilizes the limited representational capabilities at extremely low bit widths, allowing its overall performance to far surpass that of the higher-bit-width quantized BERT model. The inventors also noted that the BiBERT model lagged behind the Q-BERT model (2-8-8) and the Ternary BERT model (2-2-2) on the MNLI and STS-B tasks, but outperformed them on other tasks, suggesting that the fully binarized BERT model has greater potential for improvement on these tasks.

[0097] As shown in Tables 2 and 3, the BiBERT model provided by the inventors achieves impressive FLOPs, and requires significantly less storage space than the full-precision BERT model. Furthermore, thanks to the simple yet effective Bi-Attention mechanism, which transforms the expensive softmax function operation into a carefully designed bitwise operation BAMM, the BiBERT model surpasses other quantized BERT models in terms of computational and storage savings while maintaining the highest accuracy.

[0098] The foregoing has provided a detailed description of the neural network binarization and quantization method and apparatus for BERT models provided by this invention. Any obvious modifications made by those skilled in the art without departing from the essence of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.

Claims

1. A neural network binarization and quantization method for BERT models, used for performing natural language processing tasks, characterized in that... Includes the following steps: In the binarization stage of the BERT model, the input data first passes through the binarization embedding layer and is then fed into the converter module; the converter module includes a multi-head attention module and a feedforward network. A bidirectional attention mechanism based on information theory is introduced into the multi-head attention module, and a binary representation with maximized information entropy is adopted to enable the binary model to restore its perception of the input data. When determining the optimization direction of the BERT model, the first step is to provide a matching optimization direction by reselecting the extracted parameters. Then, a similarity matrix is ​​constructed to eliminate scale differences and numerical instability. Among them, the binarized representation maximizes the information entropy. , The probability mass function p is derived from information entropy. Random variables sampled from the middle, In the binarized linear layer, zero-mean pre-binarization weights are applied to maximize the information of the binarization weights and activations. By applying the boolean function, attention is drawn to elements with lower median weights. Binarized to 0 The bidirectional attention mechanism is implemented through the following formula: Where D is the feature dimension. These are the attention weights in binarization. It is a binary value , It is a binarized output projection. It is a binary key matrix transpose, It is a bitwise-Affine matrix multiplier, by Composed of displacement.

2. The neural network binarization and quantization method as described in claim 1, characterized in that: The input data consists of sentence numbers and sentence content from a natural language processing task, and the corresponding output data is the sentence sentiment type.

3. The neural network binarization and quantization method as described in claim 1, characterized in that... The extracted parameters are for querying. ,key Sum ;in, ； , , These represent three different binary linear layers.

4. The neural network binarization and quantization method as described in claim 1, characterized in that: Attention score Output of the multi-head attention module and hidden state The data is extracted from the full-precision teacher network in a hierarchical manner and then transferred to the binarized student network.

5. The neural network binarization and quantization method as described in claim 4, characterized in that: Mean squared error is used as the loss function to measure the difference between the student network and the teacher network on corresponding features, and then the prediction layer extracts the loss by minimizing the soft cross-entropy between the teacher's logical value and the student's logical value.

6. A neural network binarization and quantization device for BERT models, used to perform natural language processing tasks, characterized in that... Includes a processor and a memory, wherein the processor reads a computer program from the memory to perform the following operations to implement the neural network binarization and quantization method as described in any one of claims 1 to 5: In the binarization stage of the BERT model, the input data first passes through the binarization embedding layer and is then fed into the converter module. The converter module includes a multi-head attention module and a feedforward network; A bidirectional attention mechanism based on information theory is introduced into the multi-head attention module, and a binary representation with maximized information entropy is adopted to enable the binary model to restore its perception of the input data. When determining the optimization direction of the BERT model, the first step is to provide a matching optimization direction by reselecting the extracted parameters, and then a similarity matrix is ​​constructed to eliminate scale differences and numerical instability.

7. The neural network binarization and quantization device as described in claim 6, characterized in that: The input data consists of sentence numbers and sentence content from a natural language processing task, and the corresponding output data is the sentence sentiment type.

Images