FPGA-based hardware-aware differentiable BERT layer head pruning method

Abstract

The present invention discloses a hardware-aware differentiable BERT layer head pruning method based on FPGA. The method includes the following steps: introducing the idea of ​​differentiable NAS, setting an architecture parameter for each encoding layer, each attention head, and each feedforward neural network (FFN) dimension of the pre-trained BERT model; simulating a single-layer BERT model to obtain the latency and power consumption results of the four modules; calculating the latency L corresponding to the single-dimensional sub-module. s Score and power consumption P s Score; calculate L for the full model f Score and P f The method trains and updates the model's weight parameters to minimize the overall loss. After model training converges, the corresponding model structure is pruned for any values ​​below a threshold. This method can compress the BERT-base model's parameters by 1.8 times without sacrificing accuracy, while also reducing inference latency on FPGAs by 2.1 times and power consumption by 1.9 times.

Description

Technical Field

[0001] The present invention relates to the technical field of BERT layer header pruning, and in particular to a hardware-aware differentiable BERT layer header pruning method based on FPGA. Background Art

[0002] Transformer-based pre-trained models such as BERT and RoBerta have been widely used in natural language processing (NLP) tasks. These models use the attention mechanism to effectively analyze the dependencies between sequence information. However, these models typically have a large number of parameters, resulting in large inference delays and memory usage. Existing Transformer-based model pruning methods can be divided into two categories: unstructured pruning and structured pruning. In unstructured pruning, the influence of individual weights is removed by setting them to 0. However, this pruning algorithm often produces sparse weight matrices, which require further optimization of these sparse matrices before hardware deployment. Structural pruning, on the other hand, removes complete model structural blocks, such as neurons, weight matrix blocks, attention heads, or complete encoding and decoding layers.

[0003] Neural Architecture Search (NAS) is a method for automatically finding the optimal model architecture. It generally consists of three parts: a search space, a search strategy, and a performance evaluation strategy. The search space predefines the model architectures that can be discovered. The search strategy selects candidate architectures from the search space. The performance evaluation strategy evaluates the candidate architectures and provides feedback to the search strategy, allowing it to make better choices in the next iteration. Currently, commonly used search strategies are mainly based on reinforcement learning, evolutionary learning, or gradient-based differentiable search methods. Methods based on reinforcement learning and evolutionary learning are very time-consuming, with a single search requiring dozens or even hundreds of GPU days. Mainstream structural pruning methods based on Transformer models typically prune only the attention heads or only the entire encoding or decoding layers. However, a single pruning strategy cannot compress the model to its full potential, and the model may still contain some redundancy.

[0004] There are already some existing works on pruning Transformer models. However, they all currently face the following problems. The first problem is an inaccurate proxy for hardware performance. To speed up inference time, some works use FLOPs as an optimization target during training. However, FLOPs do not accurately reflect hardware performance; models with fewer FLOPs do not always have better hardware performance. The second problem is a single pruning strategy. The mainstream structural pruning methods for Transformer models are usually single attention head pruning or single encoding layer pruning. However, a single pruning strategy cannot significantly compress the model, leaving potential model redundancy and room for further optimization. The third problem is the computationally complex NAS. Current NAS-based Transformer model pruning methods typically use reinforcement learning or training the model from scratch. For researchers with limited resources, this method is computationally complex and impractical. Other works are based on differentiable NAS or NAS pruning during the fine-tuning phase. However, these methods still suffer from inaccurate hardware performance proxies and / or single pruning strategies. Summary of the Invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide an FPGA-based hardware-aware differentiable BERT layer head pruning method, which can realize structured pruning of the hybrid layer head of the BERT model, greatly compress the model while maintaining the model accuracy.

[0006] In this invention, for the complete BERT pre-trained model basic network structure before pruning, an architectural parameter is introduced for each layer, each attention head, and each FFN intermediate dimension. These architectural parameters are updated through differentiable NAS. After training convergence, by comparing these architectural parameters with a predefined threshold, the structures corresponding to the parameters less than the threshold are pruned. This part is relatively unimportant structure, thus completing the pruning of layer-head mixture based on differentiable NAS. The technical solution of the present invention is specifically described as follows.

[0007] A hardware-aware differentiable BERT layer head pruning method based on FPGA includes the following steps:

[0008] S1: Introducing the idea of ​​differentiable NAS, an architecture parameter α is set for each encoding layer, each attention head, and each feedforward neural network FFN dimension of the pre-trained BERT model. The specific steps include:

[0009] S101: First, in the pre-trained BERT model, the multi-head attention mechanism MHA is expressed as:

[0010] Multihead i=Concat(h i1 、h i2 ,...,h im )·W 0

[0011] Among them: Concat represents the concatenation operation, W o is the output linear transformation matrix, h im represents the mth attention head of the i-th encoding layer, which is defined as follows:

[0012] Q i =XW i Q ,K i =XW i K ,V i =XW i V

[0013] Where: X is input, W i Q 、W i K 、W i V are the query matrix, key matrix and value matrix of the i-th attention head respectively, and d is the dimension of the attention head;

[0014] Feedforward neural network FFN is expressed as: FFN i =max(0,W1x+b1)·W2+b2

[0015] Where: X is the input, W1, b1 are the weights and biases of the linear transformation of the first layer of FFN, and W2, b2 are the weights and biases of the linear transformation of the second layer of FFN.

[0016] The layer-by-layer model structure is represented as: Output = Layer n ·{Layer n-1 ...[Layer1(x)]},

[0017] S102: In the present invention, an architecture parameter α is set for each encoding layer of the BERT model l , set an architecture parameter α for each attention head h , set an architectural parameter α for the intermediate dimension of each FFN layer f Specifically, the BERT model after introducing the architecture parameters is expressed as follows:

[0018]

[0019] in, is the architectural parameter of the mth attention head in the i-th encoding layer, α ij f is the architectural parameter of the middle dimension of the jth FFN in the i-th coding layer, is the architectural parameter of the i-th encoding layer, and δ is the sigmoid function.

[0020] S2: Use the high-level synthesis tool Vitis HLS2020.2 to simulate the single-layer BERT model and obtain the comprehensive results of the latency and power consumption of the four modules in the single-layer BERT.

[0021] Specifically, the present invention implements the HLS code of a single-layer BERT model and simulates the embedding module, attention module, FFN feedforward neural network module, and final pooling module in the single-layer BERT. These four modules represent the embedding layer, attention layer, feedforward network layer (FFN), and final pooling layer in the BERT model, respectively. The present invention uses the Vitis HLS2020.2 advanced synthesis tool to analyze the delay, resource usage, and power consumption of the four modules, and comprehensively obtains the delay and power consumption simulation results of the four modules.

[0022] S3: Divide the delay and power consumption of the four modules simulated in S2 by the dimensions corresponding to the four modules to obtain the delay L corresponding to the single-dimensional sub-module s Score and power consumption P s Here, the subscript s refers to single, representing a single-dimensional submodule.

[0023] Specifically, for the embedding module, divide by the hidden layer dimension; for the attention module, divide by the hidden layer dimension; for the FFN feedforward neural network module, divide by the FFN intermediate layer dimension; and for the final pooling module, divide by the hidden layer dimension. Note that the hidden layer dimension and the FFN intermediate layer dimension are determined by the pre-trained model structure before pruning. For example, in the pre-trained BERT-base model before pruning, the hidden layer dimension is 768, and the FFN intermediate layer dimension is 3072.

[0024] S4: L through four unidimensional submodules s Score and P s Score, calculate the L of the complete model introduced by the searched architecture parameter α f Score and P f Here, the subscript f stands for full, representing the complete model.

[0025] Specifically, the L of the complete model is obtained by the following formula f Score and P f Fraction;

[0026]

[0027] Where: layer represents the number of layers, L s-1 、L s-2 、L s-3 、L s-4 are four unidimensional delay scores, P s-1 、P s-2 、P s-3 、P s-4 These are four single-dimensional power consumption scores. N1 and N2 are the hidden layer size and FFN middle layer size of the searched model, respectively. N1 and N2 are dynamically updated during the training process, and the search process is a training and updating process.

[0028] S5: L f Score and P f The score serves as an additional loss in addition to the cross-entropy loss during pruning training and becomes part of the optimization target, thus achieving hardware-aware pruning.

[0029] S6: Introducing a special addition Making the architecture parameter α closer to 0 or 1 reduces the performance difference between the hybrid model and the discrete model in differentiable NAS.

[0030] Where: N is the number of architecture parameters, δ is the sigmoid function;

[0031] S7: Train and update the weight parameters and architecture parameters α of the model to minimize the complete loss. Here, the complete loss includes the cross entropy loss function, L f Score and P f Score, L 0-1 Loss function:

[0032] L=L ce +L f +P f +L 0-1

[0033] The architecture parameter α obtained through training is used to complete the selection of the model structure and then realize the search of the model structure;

[0034] S8: After model training converges, for architectural parameters α that are less than a threshold, the corresponding model structures are pruned. The pruned structures may include the complete encoding layer, attention head, and FFN intermediate dimensions. This is the pruning method of the present invention, which is a layer-head hybrid pruning method.

[0035] Specifically, after the model training converges, the value of each architecture parameter α is compared with the predefined threshold τ. When the parameter is less than the threshold τ, it is judged that the structure corresponding to this parameter is not important and pruning is performed. Then the i-th encoding layer can be pruned during inference; if Then the mth attention head of the i-th encoding layer can be pruned during inference; if Then, during inference, the j-th FFN intermediate dimension of the i-th coding layer can be pruned, and finally the layer head mixing pruning is completed.

[0036] In the above step S6, the present invention introduces a special 0-1 loss function Where N is the number of architecture parameters and δ is the sigmoid function. By adding this loss function, the penalty for architecture parameters is minimized when they are 0 or 1, which reduces the structural difference between the hybrid model and the pruned discrete model during the differentiable NAS search process.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention directly introduces the hardware-aware inference delay evaluation score L based on the hardware-aware model evaluation index of FPGA HLS. f And the inference power consumption evaluation score P f ,These two scores are obtained by modeling and calculating by performing HLS high-level synthesis simulation on the target FPGA. f Score and P f The score directly reflects the latency and power consumption of the model inference and can be optimized directly during model training. It can more accurately represent the hardware performance of the model than FLOPs.

[0039] This paper implements hybrid layer head pruning for the Transformer model based on differentiable NAS. This paper uses differentiable NAS to automatically search for the optimal model architecture, finding the best set of encoding layers and attention heads, rather than a single head or layer-by-layer compression, significantly compressing the model.

[0040] This paper introduces a special 0-1 loss function to the pruning training process based on differentiable NAS, which makes the architectural parameter α closer to a 0-1 distribution. In this way, the present invention narrows the structural gap between the hybrid model searched in differentiable NAS and the pruned discrete model. Experiments on the GLUE benchmark dataset show that compared to the baseline BERT, the present invention achieves a 2.1x speedup, a 1.9x power reduction, and a 1.8x reduction in model size without any loss in accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flowchart of the present invention's FPGA-based hardware-aware differentiable BERT layer head pruning method.

[0042] Figure 2Detailed diagram of a differentiable BERT layer head pruning method proposed in this paper. (a) The complete BERT base network structure before pruning. (b) An architectural parameter is introduced for each layer, each attention head, and each intermediate dimension of the FFN, and these architectural parameters are updated using differentiable NAS. Lighter colors indicate smaller parameters and less important structures. (c) Irrelevant structures are pruned.

[0043] Figure 3 This is a schematic diagram of the special 0-1 loss function L0-1 introduced in the present invention. DETAILED DESCRIPTION

[0044] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0045] In the description of the present invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front end," "rear end," "both ends," "one end," "the other end," and the like, indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limiting the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0046] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "provided with," "connected," etc., should be understood in a broad sense. For example, "connected" may refer to a fixed connection, a detachable connection, or an integral connection; it may refer to a mechanical connection or an electrical connection; it may refer to a direct connection or an indirect connection through an intermediate medium; it may refer to internal communication between two components. Those skilled in the art will be able to understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0047] Please refer to Figure 1-3 As shown, the present invention is a hardware-aware differentiable BERT layer head pruning method based on FPGA, comprising the following steps:

[0048] S1: Introducing the idea of ​​differentiable NAS, an architectural parameter α is set for each encoder layer, each attention head, and each feedforward neural network (FFN) dimension of the pre-trained BERT model. The specific steps include:

[0049] S101: First, in the pre-trained BERT model, the multi-head attention mechanism MHA is expressed as:

[0050] Multihead i =Concat(h i1 、h i2 ,...,h im )·W 0

[0051] Among them: Concat represents the concatenation operation, W o is the output linear transformation matrix, h im represents the mth attention head of the i-th encoder layer, which is defined as follows:

[0052] Q i =XW i Q ,K i =XW i K ,V i =XW i V

[0053] Where: X is input, W i Q 、W i K 、W i V are the query matrix, key matrix and value matrix of the i-th attention head respectively, and d is the dimension of the attention head;

[0054] Feedforward neural network FFN is expressed as: FFN i =max(0,W1x+b1)·W2+b2

[0055] Where: X is the input, W1, b1 are the weights and biases of the linear transformation of the first layer of FFN, and W2, b2 are the weights and biases of the linear transformation of the second layer of FFN.

[0056] The layer-by-layer model structure is represented as: Output = Layer n ·{Layer n-1 ...[Layer1(x)]},

[0057] S102: In the present invention, an architecture parameter α is set for each encoding layer of the BERT model l , set an architecture parameter α for each attention head h , set an architectural parameter α for the intermediate dimension of each FFN layer f Specifically, the BERT model after introducing the architecture parameters is expressed as follows:

[0058]

[0059] in, is the architectural parameter of the jth attention head in the i-th layer, α ij f is the architectural parameter of the intermediate dimension of the j-th FFN in the i-th layer, is the architectural parameter of the i-th layer, and δ is the sigmoid function.

[0060] S2: Use the high-level synthesis tool Vitis HLS2020.2 to simulate the single-layer BERT model and obtain the comprehensive results of the latency and power consumption of the four modules in the single-layer BERT.

[0061] Specifically, the present invention implements the HLS code of a single-layer BERT model and simulates the embedding module, attention module, FFN feedforward neural network module, and final pooling module in the single-layer BERT. These four modules represent the embedding layer, attention layer, feedforward network layer (FFN), and final pooling layer in BERT-base, respectively. The present invention uses the VitisHLS2020.2 advanced synthesis tool to analyze the delay, resource usage, and power consumption of the four modules, and comprehensively obtains the delay and power consumption simulation results of the four modules.

[0062] S3: Divide the delay and power consumption of the four modules simulated in S2 by the dimensions corresponding to the four modules to obtain the delay L corresponding to the single-dimensional sub-module s Score and power consumption P s Here, the subscript s refers to single, representing a single-dimensional submodule.

[0063] Specifically, for the embedding module, divide by the hidden layer dimension; for the attention module, divide by the hidden layer dimension; for the FFN feedforward neural network module, divide by the FFN intermediate layer dimension; and for the final pooling module, divide by the hidden layer dimension. Note that the hidden layer dimension and the FFN intermediate layer dimension are determined by the pre-trained model structure before pruning. For example, in the pre-trained BERT-base model before pruning, the hidden layer dimension is 768, and the FFN intermediate layer dimension is 3072.

[0064] S4: L through four unidimensional submodules s Score and P s Score, calculate the L of the complete model f Score and P f Here, the subscript f stands for full, representing the complete model.

[0065] Specifically, the L of the complete model is obtained by the following formula f Score and P f Fraction;

[0066]

[0067] Where: layer represents the number of layers, L s-1 、L s-2 、L s-3 、L s-4 are four unidimensional delay scores, P s-1 、P s-2 、P s-3 、P s-4 are four single-dimensional power consumption scores, N1 and N2 are the hidden layer size and FFN intermediate layer size of the searched model respectively.

[0068] For example, the BERT pre-trained model (BERT-base model) contains 12 layers, each with 12 attention heads. The hidden layer size and FFN intermediate layer size of the BERT pre-trained model are 768 and 3072 respectively.

[0069] S5: L f Score and P f The score serves as an additional loss in addition to the cross-entropy loss during pruning training and becomes part of the optimization target, thus achieving hardware-aware pruning.

[0070] S6: Introducing a special addition Making the architecture parameter α closer to 0 or 1 reduces the performance difference between the hybrid model and the discrete model in differentiable NAS.

[0071] Where: N is the number of architecture parameters, and δ is the sigmoid function.

[0072] S7: Train and update the weight parameters and architecture parameters α of the model to minimize the complete loss. Here, the complete loss includes the cross entropy loss function, L f Score and P f Score, L 0-1 Loss function:

[0073] L=L ce +L f +P f +L 0-1

[0074] S8: After model training converges, for architectural parameters α that are less than a threshold, the corresponding model structures are pruned. The pruned structures may include the complete encoder layer, attention head, and FFN intermediate dimensions. This is the pruning method of the present invention, which is a layer-head hybrid pruning method.

[0075] Specifically, after the model training converges, the value of each architecture parameter α is compared with the predefined threshold τ. When the parameter is less than the threshold τ, it is judged that the structure corresponding to this parameter is not important and pruning is performed. Then the i-th layer can be pruned during inference; if Then the j-th attention head of the i-th layer can be pruned during inference; if Then, during inference, the j-th FFN intermediate dimension of the i-th layer can be pruned, and finally the layer head mixed pruning is completed.

[0076] In the above step S1, differentiable NAS refers to differentiable neural network architecture search. NAS (Nerual Architecture Search) is an automated method for finding the best model architecture. It usually consists of three parts: search space, search strategy, and performance evaluation strategy. The search space presupposes the architecture that can be discovered in principle. The search strategy selects candidate architectures from the search space. Then, the performance evaluation strategy evaluates the candidate architectures and feeds back to the search strategy to guide better choices in the next iteration. Current search methods are mainly based on reinforcement learning, evolutionary learning, and gradient-based differentiable NAS search. Reinforcement learning-based and learning-based methods can be time-consuming, lasting tens or hundreds of GPU days. In contrast, differentiable NAS achieves gradient-based optimization by continuously relaxing the search space, greatly shortening the search process.

[0077] In step S2 above, the high-level synthesis tool provides a corresponding framework for Vitis HLS, the Vitis application acceleration development process, which can be used to develop and deliver FPGA acceleration applications for software and hardware components using standard programming languages. Software components or host programs are developed in C / C++ and can run on x86 or embedded processors, using OpenCL or XRT native API calls to manage runtime interactions with the accelerator. Hardware components or kernels (running on actual FPGA cards / platforms) can be developed in C / C++, OpenCL C or RTL. The Vitis software platform helps facilitate the concurrent development and testing of hardware and software elements of heterogeneous applications. Therefore, software programs running on the host need to use precisely defined interfaces and protocols to communicate with the acceleration kernels running on the FPGA hardware model

[0078] In the above step S6, the present invention introduces a special 0-1 loss function Where N is the number of architecture parameters and δ is the sigmoid function. By adding this loss function, the architecture parameters are penalized to the minimum when they are 0 or 1, which reduces the structural difference between the hybrid model and the pruned discrete model during the differentiable NAS search process. For example, when pruning the attention head, when the 0-1 loss function is not introduced, the hybrid model is expressed as: Multihead i =Concat(0.385h il +0.955h i2 +0.997h i3 )W 0 , when the threshold is 0.99, the pruned discrete model is represented as: Multihead i =Concat(h i2 +h i3 )W 0 , so the structural differences are due to:

[0079] Gap=Concat(-0.385h i1 +0.005h i2 +0.003h i3 )W 0 But after applying 0-1 loss, since the architecture parameters are more likely to be 0 or 1, the searched hybrid model is expressed as:

[0080] Multihead i =Concat(0.005h il +0.999h i2 +0.999h i3 )W 0 , the pruned discrete model is represented as: Multihead i =Concat(h i2 +h i3 )W 0 , the structural difference is given by Gap=Concat(-0.005h i1 +0.001h i2 +0.001h i3 )W 0 After the introduction of the 0-1 loss, the structural difference between the searched hybrid model and the pruned discrete model becomes smaller. Therefore, the performance gap between the two models is narrowed, which helps to guide a more efficient search process during pruning training.

[0081] In the above step S8, during the hybrid layer head pruning process of the present invention, the basic network structure of the complete BERT pre-trained model before pruning is introduced, and an architectural parameter is introduced for each layer, each attention head, and each FFN intermediate dimension. These architectural parameters are updated through differentiable NAS. After the training converges, by comparing the size of these architectural parameters with the predefined threshold, the structure corresponding to the parameters smaller than the threshold is pruned. This part is a relatively unimportant structure, thereby completing the pruning of the layer head mixture based on differentiable NAS.

[0082] Experiments were conducted on the GLUE benchmark dataset. Compared with the baseline BERT, the present invention achieved 2.1 times acceleration, 1.9 times power consumption reduction, and 1.8 times model size reduction without loss of accuracy.

[0083] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A hardware-aware differentiable BERT layer head pruning method based on FPGA, characterized by: The following steps are involved: S1: Introducing the idea of ​​differentiable NAS, setting an architectural parameter α for each encoding layer, each attention head, and each feedforward neural network FFN dimension of the pre-trained BERT model; S2: Simulate a single layer in the pre-trained BERT model to obtain the comprehensive results of the latency and power consumption of the four modules in the single-layer BERT model; the four modules are the embedding module, the attention module, the feed-forward neural network FFN module, and the final pooling module. These four modules represent the embedding layer, attention layer, feed-forward network FFN layer, and final pooling layer in the pre-trained BERT model respectively; S3: Divide the latency and power consumption of the four modules simulated in S2 by the dimensions corresponding to the four modules in the pre-trained BERT model to obtain the latency fraction L corresponding to the single-dimensional sub-module s and power consumption fraction P s , where the dimensions of the embedding module, attention module, and final pooling block in the pre-trained BERT model to be pruned are the hidden layer dimensions, and the dimension of the FFN feedforward neural network module is the FFN intermediate layer dimension; S4: L through four unidimensional submodules s Score and P s Score, calculate the hardware-aware inference latency evaluation score L of the BERT model that introduces the architecture parameter α f And the inference power consumption evaluation score P f : Where: layer represents the number of layers, L s-1 、L s-2 、L s-3 、L s-4 is the delay fraction of the four unidimensional submodules, P s-1 、P s-2 、P s-3 、P s-4 is the power consumption fraction of the four single-dimensional sub-modules, N1 and N2 are the hidden layer size and FFN intermediate layer size of the BERT model with the architecture parameter α. S5: Training updates the weights and architecture parameters α of the BERT model that introduces the architecture parameter α, minimizing the complete loss function L of the model, which includes the cross entropy loss function L ce , L f Score, P f Score and loss function L 0-1 : L=L ce +L f +P f +L 0-1 Where: loss function L 0-1 It is used to make the architecture parameter α closer to 0 or 1, reduce the performance difference between the hybrid model searched in the differentiable NAS and the discrete model after pruning, and guide the search, that is, the training update process; S6: After the BERT model training with the architecture parameter α converges, the corresponding model structure for the architecture parameter α that is less than the threshold τ is pruned to achieve mixed layer head pruning of the BERT model; the pruned model structure is selected from one or more of the encoding layer, attention head, or FFN intermediate dimension.

2. The hardware-aware differentiable BERT layer head pruning method according to claim 1, characterized in that: In step S1, the multi-head attention mechanism MHA is expressed as follows in the BERT model with the architecture parameter α: Among them: Concat represents the concatenation operation, W o is the output linear transformation matrix, h im represents the mth attention head of the i-th encoding layer, is the architectural parameter of the mth attention head in the i-th encoding layer, and δ is the sigmoid function; The feedforward neural network FFN is expressed as: Where: x is the input, W1, b1 are the weights and biases of the linear transformation of the first layer of FFN, W2, b2 are the weights and biases of the linear transformation of the second layer of FFN, α i f is a j-dimensional parameter, j is the dimension of the FFN layer in the i-th layer, α i f Represents the architectural parameters of the intermediate dimension of the FFN layer in the i-th layer; The model structure by layer is expressed as: Where, Layer represents the architecture parameters of the i-th coding layer, i = 1, 2, 3...n, i (x) represents the i-th coding layer.

3. The hardware-aware differentiable BERT layer head pruning method according to claim 1, characterized in that: In step S1, the pre-trained BERT model is the BERT-base model.

4. The hardware-aware differentiable BERT layer head pruning method according to claim 1, characterized in that: In step S2, the high-level synthesis tool Vitis HLS is used to simulate a single layer in the pre-trained BERT model.

5. The hardware-aware differentiable BERT layer head pruning method according to claim 1, characterized in that: In step S5, the loss function L 0-1 It is expressed as follows: Where: α is the architecture parameter, N is the number of architecture parameters, and δ is the sigmoid function.

6. The hardware-aware differentiable BERT layer head pruning method according to claim 2, characterized in that: In step S5, if Prune the i-th coding layer; if Prune the mth attention head of the i-th encoding layer; if Prune the jth FFN intermediate layer dimension of the i-th coding layer, and finally complete the pruning of the layer head mixture, α ij f Represents the architectural parameters corresponding to the j-th FFN intermediate layer dimension of the i-th coding layer.

Images