Pre-training language model compression method and mobile terminal deployment method

Abstract

The present disclosure relates to a pre-training language model compression method and a mobile terminal deployment method. The compression method comprises: taking an original PLM as a super network, selecting n blocks with different widths from a target function layer of the super network to construct a sub-network set; setting a mask layer of n different width differentiable mask paths after the target function layer; calculating the aggregated output of the target function layer for the n different mask paths; based on the aggregated output, training the super network according to a predetermined penalty function to adjust the network parameters; performing network architecture search on the trained super network to determine a compressed target network with optimized performance from the sub-network set. By taking the original PLM as a super network and training using a differentiable mask technique, the present invention avoids the overfitting problem during model compression. The above differentiable mask super network training can be combined with mask path-based search space pruning, greatly reducing the resource requirements during subsequent evolutionary search.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning, and in particular to a method for compressing a pre-trained language model and a method for deploying it on mobile devices. Background Technology

[0002] Due to the effectiveness of self-attention or cross-attention mechanisms, the Transformer and its BERT model, which utilizes the Transformer, possess excellent relevance and dynamic modeling capabilities, and are widely used in information processing of language, speech, and even visual modalities, demonstrating outstanding performance in Natural Language Processing (NLP). Self-supervised pre-training and downstream task-based fine-tuning of the BERT model have become the mainstream application paradigm, significantly improving the performance of NLP tasks. However, the large number of parameters and FLOPS (floating-point operations per second) of attention-based models hinder their deployment on resource-constrained devices, greatly limiting the application areas of these models.

[0003] Therefore, a solution is needed that can deploy pre-trained language models on resource-rich devices. Summary of the Invention

[0004] One technical problem this disclosure aims to solve is to provide a pre-trained language model compression method and a mobile deployment method. By using the original PLM as a supernetwork and training it using differentiable masking techniques, the overfitting problem during model compression is avoided. The above-mentioned differentiable masked supernetwork training can be further combined with search space pruning based on masked paths, thereby significantly reducing the resource requirements during subsequent evolutionary search.

[0005] According to a first aspect of this disclosure, a method for compressing a pre-trained language model is provided, comprising: using the original pre-trained language model as a hypernetwork of the target model, and selecting layers of different widths from the target function layer of the hypernetwork. n Each block is used to construct a set of subnetworks, where... n It is an integer not less than 2; a mask layer is set after the objective function layer of the hypernetwork, wherein the mask layer includes n Differentiable mask paths of different widths are used to mask the output of the objective function layer with different widths, wherein the masks... n Each different width corresponds to one of the subnetwork sets. n Blocks of different widths; the objective function layer is obtained for... nAggregate the output of different masked paths; based on the aggregated output, train the supernetwork according to a predetermined penalty function to adjust the parameters of the supernetwork; perform network architecture search on the trained supernetwork to determine a compressed target network with optimized performance from the set of subnetworks.

[0006] Optionally, the method further includes: after the supernetwork training is completed, obtaining the importance ranking of the mask paths according to the distribution of the differentiable structure parameters of each block, and removing those paths whose importance ranking is not high. k The mask path of each, where k It is less than n The positive integer, and the network architecture search is performed on pruned supernetworks.

[0007] Optionally, the original BERT is composed of multiple identical stacked unit structures, each unit structure including a multi-head attention (MHA) structure and a feedforward network (FFN) structure, and the objective function layer of the supernetwork includes: a splicing layer of the MHA structure, and a mask layer set after the splicing layer to control the number of heads of the MHA structure; and / or a first linear layer of the FFN structure, and a mask layer set after the first linear layer to control the intermediate size of the FFN structure.

[0008] Optionally, performing network architecture search on the trained supernetwork includes performing a resilient structure search in a search space along at least one of the following dimensions: the number of heads in the MHA structure; the intermediate size of the FFN structure; and the network depth.

[0009] Optionally, for the same objective function layer, a wider block encompasses all the output elements contained in a narrower block.

[0010] Optionally, during the training of the supernetwork, for each mini-batch of training data, a portion of the neuron nodes are randomly discarded at a predetermined discard rate.

[0011] Optionally, randomly discarding a portion of neuron nodes at a predetermined discard rate includes: assigning a predetermined discard rate adapted to the width of the mask path to different mask paths.

[0012] Optionally, the method further includes: using the original pre-trained language model as a teacher model to fine-tune the compressed target network based on knowledge distillation.

[0013] Optionally, using the original pre-trained language model as the teacher model, fine-tuning the compressed target network based on knowledge distillation includes: using a weighted average of target class knowledge distillation and non-target class knowledge distillation to fine-tune the compressed target network.

[0014] Optionally, evolutionary search can be used to search for the network architecture of the trained supernetwork. The accuracy of downstream tasks can be used as a selection criterion for individual models in the evolutionary iteration, and the iteration can be stopped when the accuracy reaches a predetermined threshold, and the individual model with the highest accuracy can be selected as the target model.

[0015] According to a second aspect of this disclosure, a method for deploying a pre-trained language model on a mobile device is provided, comprising: deploying a target network obtained according to the first aspect on a mobile device; and the mobile device using the target network to perform downstream natural language processing tasks.

[0016] According to a third aspect of this disclosure, a computing device is provided, comprising: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method described in the first aspect above.

[0017] According to a fourth aspect of this disclosure, a computer program product is provided, including executable code that, when executed by a processor of an electronic device, causes the processor to perform the method described in the first aspect above.

[0018] According to a fifth aspect of this disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored, which, when executed by a processor of an electronic device, causes the processor to perform the method described in the first aspect above.

[0019] Therefore, to facilitate the deployment of PLM models, especially BERT models, on resource-constrained devices, a resilient single-pass NAS framework for multi-size BERT compression is proposed, supplemented by a pipeline consisting of differentiable masks, search space pruning, and evolutionary search in the preferred implementation. Experimental results demonstrate that complex BERT models can be effectively compressed using the proposed method, indicating significant potential for industrial applications. Attached Figure Description

[0020] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0021] Figure 1 An overview of the classic neural architecture search is shown.

[0022] Figure 2 A method for compressing a pre-trained language model according to an embodiment of the present invention is shown.

[0023] Figure 3A -C shows an example used to illustrate the attention mechanism.

[0024] Figure 4 An example of a differentiable mask along the width of the network is shown.

[0025] Figure 5 A multi-head attention module with DMasking is shown.

[0026] Figure 6 The compression process of a pre-trained language model according to a preferred embodiment of the present invention is illustrated.

[0027] Figure 7 An example of using DMasking for search space pruning is shown.

[0028] Figure 8 A compression model system according to an embodiment of the present invention is shown.

[0029] Figure 9 A schematic diagram of a computing device that can be used to implement the above-described pre-trained language model compression method according to an embodiment of the present invention is shown. Detailed Implementation

[0030] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0031] BERT (Bidirectional Encoder Representation from Transformers) is a pre-trained language representation model that no longer uses traditional unidirectional language models or shallow concatenation of two unidirectional language models for pre-training. Instead, it uses a new MLM (Mask Language Model) which can generate deep bidirectional language representations.

[0032] The computationally intensive nature of BERT-like models hinders their deployment on resource-constrained platforms. To effectively reduce the memory footprint and computational complexity of the Transformer sub-models in BERT models while maintaining acceptable accuracy, model compression has become a mainstream research direction, mainly including lightweight architecture design, model quantization, and Neural Architecture Search (NAS). For specific downstream tasks, based on the architecture of pre-trained BERT models, resilient NAS can be performed during downstream propagation with multi-scale compression along width and depth.

[0033] Figure 1 This diagram provides an overview of classic neural architecture search. As shown, the classic NAS method uses an RNN as a controller to generate subnetworks, then trains and evaluates these subnetworks to obtain their network performance (e.g., accuracy), and finally updates the controller parameters. The search needs to be performed in a dedicated space, which is the set of candidate network structures to be searched. The complexity of the search space determines the potential of the network structure. The search also requires a specific search strategy, i.e., how to select from the search space. Depending on the method, search strategies are broadly categorized into reinforcement learning-based methods, evolutionary algorithm-based methods, and gradient-based methods. The network structures obtained based on the search strategy need performance evaluation, i.e., evaluating the performance of the network structures on the target dataset. The simplest method is to train each subnetwork on the training data and evaluate its performance on the test data; however, training so many structures from scratch is too time-consuming. Weight sharing can be used instead of re-initialization, significantly accelerating the search process. Alternatively, performance evaluation can be performed by using a small portion of the dataset for evaluation, reducing network parameters, training fewer epochs, or predicting the trend of network training.

[0034] One-shot structure search is currently the mainstream approach. This method defines the search space as a supernet, encompassing all network structures. The search is then performed within a large, overparameterized network, alternating between training network and model weights, ultimately retaining only one substructure. Existing resilient NAS methods require extensive supernet training and subnet sampling to ensure effective weight sharing, resulting in relatively high training costs. Furthermore, downstream datasets are generally small, and long supernet training times can easily lead to overfitting in the BERT model, resulting in poor prediction accuracy in the compressed model.

[0035] To address this, this invention proposes a resilient single-pass neural architecture search (NAS) framework for BERT compression. It enhances the training of the supernetwork for downstream tasks by introducing DMasking (differentiable masking), a training process that can be further refined through adaptive dropout. The search space can be pruned based on the importance ranking of differentiable structural parameters to reduce the overhead of subsequent evolutionary searches. Furthermore, improved knowledge distillation can be introduced to ensure the accuracy of the compressed model. Experiments on benchmarks demonstrate that the compressed model using the proposed method achieves better results than strong baselines. In particular, the extremely compressed BERT with 15M stacked BERT layer parameters achieves nearly 2.8x compression on the MNLI task with negligible accuracy loss.

[0036] Figure 2A pre-trained language model compression method according to an embodiment of the present invention is illustrated. This method uses the original pre-trained language model (e.g., a pre-trained BERT) as a supernetwork and performs a single NAS on this supernetwork to find a subset with optimized performance as the target network. Unlike conventional NAS methods, the present invention avoids overfitting by introducing DMasking.

[0037] Specifically, in step S210, the original pre-trained language model is used as the hypernetwork of the target model, and different widths are selected from the objective function layer of the hypernetwork. n Each block is used to construct a set of subnetworks, where... n It is an integer not less than 2. Here, the objective function layer is a layer that reduces computational cost while maintaining correct network operation, even with a reduced width. The standard BERT structure is defined as... A stack of identical transformer-encoder layers, each consisting of a multi-head attention (MHA) structure and a feedforward network (FFN) structure. When using BERT as the supernetwork, the objective function layer can be a concatenation layer in the MHA structure and / or the first linear layer in the FFN structure.

[0038] To facilitate understanding of the principles of this invention, the following will be combined with... Figure 3A -C introduces the structure of MHA. Figure 3A -C shows an example used to illustrate the attention mechanism. Specifically, Figure 3A This illustrates the application of the attention mechanism in image-to-text tasks. During the encoding process, an encoder, such as one implemented using a CNN (Convolutional Neural Network), encodes the input pattern into an embedding vector. This embedding vector is then fed into a decoder consisting of an LSTM (Long Short-Term Memory) network and an attention mechanism. As shown on the right side of the image, "A bird flying over a body of water" is a description of the image on the right. The differences in brightness in the three middle images reflect the magnitude of the attention weights assigned to corresponding parts by the attention mechanism in the decoder; brighter areas correspond to higher weights. It can be seen that when obtaining "A" (a bird), more attention is paid to the position corresponding to the bird's wing in the image (i.e., this position has a higher weight); when obtaining "bird," more attention is paid to the position corresponding to the bird in the image; and when obtaining "water," more attention is paid to the position corresponding to the body of water in the image. The attention mechanism has a significant advantage in terms of the interpretability of deep learning, making the process of obtaining the final output of the model more consistent with human intuitive cognition.

[0039] The self-attention mechanism directly calculates the attention weights at each position of the input (in BERT, the input is a sentence) during the encoding process through some operation; then, the implicit vector representation of the entire sentence is calculated in the form of a sum of weights. Figure 3B A structural diagram of the self-attention mechanism is shown. As can be seen from the diagram, the core process of the self-attention mechanism is through… Q and K The attention weights are calculated; then applied to V Obtain the total weights and output. Q , K and V Each by input With matrix , , The result is obtained by multiplication. The result is obtained by calculation. Q , K and V ( Q , K and V These correspond to Query, Key, and Value respectively, which can be understood as referring to the same input. Three different linear transformations are performed to represent its three different states. After obtaining... Q , K and V Then, the weight vector can be calculated further.

[0040] When encoding information about the current position, the model may overemphasize its own location while neglecting other locations. To address this, multi-head attention (MHA) can be employed. Furthermore, MHA allows the attention layer's output to contain encoded representations from different subspaces, thereby enhancing the model's expressive power. Figure 3C The diagram illustrates the structure of a self-attention mechanism. As can be seen from the diagram, a multi-head attention mechanism involves performing multiple sets of self-attention processes on the original input sequence; then, the results of each set of self-attention are concatenated and subjected to a linear transformation to obtain the final output. Alternatively, the multi-head mechanism can be viewed as dividing a high-dimensional single-head sequence into... Size.

[0041] As mentioned earlier, the standard BERT architecture is defined as follows: A stack of identical transformer-encoder layers. Here, we can assume that the... BERT layers ( <= The input is ,in It is the sequence length and It is the size of the embedding vector. The calculation of MHA is divided into... One of the heads From the matrix , , Parameterization, where Therefore, the head is deduced. Attention map:

[0042] The encoded embedding vector of MHA is obtained by concatenating multi-head attention and linear projection.

[0043] In a feedforward neural network, each neuron belongs to a different layer. Neurons in each layer receive signals from neurons in the previous layer and output signals to the next layer. Layer 0 is called the input layer, the last layer is called the output layer, and the other intermediate layers are called hidden layers. There is no feedback in the entire network; signals propagate unidirectionally from the input layer to the output layer. The FFN used in the transformer-encoder layer has two linear layers. These two linear layers are composed of… , and , Parameterization, where This is the intermediate size of the FFN. Assume the input to the FFN is... Therefore, the output embedding vector after two layers of FFN can be determined as:

[0044] The network width is initially defined as the number of heads in the MHA or the intermediate size of the FFN. To reduce the complexity of BERT, a flexible single-pass NAS can be implemented to compress BERT along both width and depth within the search space to obtain multiple compressed subnetworks.

[0045] Specifically, from having Layer stack ( <= And is represented as The subnetwork extracted from BERT can be represented by the following formula:

[0046] in It is the first The sampling submodule within MHA or FFN in each BERT layer These are the corresponding weights. Each layer has multiple block selections of different widths, i.e., in... Choose from a set of candidates, as shown below:

[0047] in It is the first Candidate modules within the layer, It is a weight. The elastic structure selection described in equation (4) indicates all in the same layer The choices share weights. For the same objective function layer, a wider block encompasses all the output elements contained in a narrower block. That is, for the , Any two submodules in the layer and The weights involved can be left-aligned:

[0048] In summary, BERT, as a supernet, can be compressed through elastic search along its width and depth, mainly consisting of two stages: supernet training and discrete architecture search (such as evolutionary search). During supernet training, subnetworks are uniformly sampled at each step, and shared weights are updated. Therefore, sufficient supernet training requires a long time, making BERT prone to overfitting.

[0049] To overcome overfitting during elastic single NAS, differentiable masking (DMasking) along the network width is used to enhance supernet training. To this end, in step S220, a mask layer is placed after the objective function layer of the supernet, wherein the mask layer corresponds to... n Differentiable mask paths of different widths are used to mask the output of the objective function layer with different widths, wherein the masks... n Each different width corresponds to one of the subnetwork sets. n In step S230, the objective function layer is calculated for blocks of different widths. n The aggregated output of several different mask vectors. In step S240, the supernetwork can be trained based on the aggregated output according to a predetermined penalty function to adjust the parameters of the supernetwork.

[0050] Figure 4 An example of a differentiable mask along the network width is shown. As illustrated, four mask vectors are placed after the output of the function layer. ( The subpath of ), where the size of the corresponding width is represented as And the differentiable structure parameters are expressed as ( ).exist Figure 4 In the example,n The value is 4, and the mask vector m 1. Only one-quarter of the parameters of the illustrated function layer are retained; that is, three-quarters of the parameters are masked. When the objective function layer is a splicing layer in the MHA structure, if the MHA computation is divided into... The head, in the mask vector m 1. Mask 6 out of the 8 headers so that the first masked path includes only 2 headers. Similarly, the mask vector... m 2. Only the parameters of half of the illustrated function layer are retained; that is, the parameters of half are masked. When the objective function layer is a splicing layer in the MHA structure, if the MHA computation is divided into... The head, in the mask vector m 2. Mask four of the eight headers so that the second mask path includes all four headers. Mask vector m 3. Retain 3 / 4 of the parameters in the illustrated function layer; that is, mask 1 / 4 of the parameters. When the objective function layer is a concatenation layer in the MHA structure, if the MHA computation is divided into... The head, in the mask vector m Mask two of the eight headers so that the third mask path includes six headers. Mask vector m 4. Retain all parameters of the illustrated function layer; that is, do not perform masking. In the case where the objective function layer is a concatenation layer in the MHA structure, if the MHA computation is divided into... The head, in the mask vector m The fourth mask path corresponding to 4 includes the complete structure of the objective function layer. It can be understood that if the objective function layer is the first layer of an FFN, the mask vector masks the intermediate dimensions of the first linear layer output, corresponding to 1 / 4, 1 / 2, 3 / 4, and the entire original intermediate dimensions, respectively. It should also be understood that the parameters preserved in the mask vector can be left-aligned.

[0051] Additionally, due to the mask's n Each different width corresponds to one of the subnetwork sets. n The target function layer is divided into blocks of varying widths, resulting in four blocks with heads of 2, 4, 6, and 8 respectively in the constructed subnetwork set. DMasking aggregates the outputs from different masking paths. In other words, after adding the masking layer, the original output of the target function layer becomes the output of the target function layer. n The aggregated output under each masked path. Therefore, the aggregated output of the function layer with DMasking can be represented as follows (in this example) ):

[0052] in It is input The primitive function. It is a weighted mixture of mask vectors. By default, there are conditions: . For differentiable structure parameters The Gumbel-Softmax function performs the operation. In this invention, the resource budget penalty can be applied based on FLOPS or model size.

[0053] For elastic search along the network width, the function layer in equation (6) is either the attention graph in MHA or the first layer in FFN. Figure 5 A multi-head attention (MHA) module with DMasking is shown. Figure 5 As shown, the DMasking function is placed after the concatenation layer of the split attention map to flexibly select the number of heads, where the equivalent computation of attention with DMasking is expressed as:

[0054] in It is the first Barcode path ( The number of heads (=1,2,3,4).

[0055] As mentioned earlier, the multi-head mechanism can be considered as dividing a high-dimensional single head into... In this invention, the addition of the DMasking function enables the control of multiple heads. The selection process involves filtering by size; for example, in an optimized compression model, only specific layers are ultimately selected for that layer. Size. Additionally, it should be noted that each mask path corresponding to the DMasking function in the diagram is left-aligned, which ensures that effective information during supernet training is uniformly concentrated in the parameters on the left.

[0056] DMasking can also be placed between two layers within an FFN to flexibly define an intermediate size. Therefore, the output embedding vector of an FFN with DMasking is calculated as follows:

[0057] The benefits of DMasking are: i) the parallel placement of multiple mask paths makes supernet training more thorough and efficient than sampling-based methods, and helps avoid overfitting; ii) through the leftmost-preserving scheme, the mask along the network width is actually used for BERT compression, where the dense layers involved can be pruned for deployment, and these dense layers include those in MHA. The projection layer and two layers in FFN.

[0058] During the training of a supernetwork, DMasking can be randomly dropped out, especially adaptively, thereby reducing the risk of overfitting and enhancing generalization ability.

[0059] Dropout refers to randomly discarding a portion of neurons with a certain probability p. This "discarding" is only temporary, occurring with each mini-batch of training data. Because it's random, the neural network structure is different each time, meaning each iteration trains a neural network with a different structure. Specifically, during training, the activation values ​​of neurons are "discarded" with a certain probability p, essentially "stopping" the process. Therefore, for a network with N neurons, dropout can be seen as generating 2... N This process, which involves creating a set of models, weakens the joint adaptation among all neurons, reduces the risk of overfitting, and enhances generalization ability.

[0060] In this invention, dropout is utilized during the training of the supernet using DMasking for regularization purposes. To this end, the compression method of this invention includes, during the training of the supernet, randomly dropping a portion of neurons at a predetermined dropout rate for each mini-batch of training data. If the same dropout rate is applied directly to all mask paths, subnetworks with different widths will be trained unfairly. Therefore, a predetermined dropout rate adapted to the width of the mask path can be assigned to different mask paths. In one embodiment, to prevent overfitting of large subnetworks and underfitting of small subnetworks, the adaptive dropout along the network width is designed as follows:

[0061] in Assigned as the discard rate to the first Mask path, It is the size of the corresponding mask width, used to control the decay ratio.

[0062] It should be understood that training a hypernetwork using DMasking requires iterations. In the implementation of this invention, it was found that several epochs of training are needed to obtain good compression results downstream.

[0063] After the supernetwork training is completed, in step S250, a network architecture search is performed on the parameter-tuned supernetwork to determine the compressed target network with optimized performance from the set of subnetworks. The accuracy of downstream tasks can be used as a selection criterion for individual models in the evolutionary iteration, and iteration stops when the accuracy reaches a predetermined threshold, selecting the individual model with the highest accuracy as the target model. Corresponding to the above description, the network architecture search for the parameter-tuned supernetwork can also be performed on two dimensions: the number of heads in the MHA structure and the median size of the FFN structure. In one embodiment, a flexible structure search can also be performed on the search space based on the network depth dimension. In this case, when compressing the BERT model, the target model no longer needs to have a specific structure. l Instead of one transformer-encoder layer, it has fewer transformer-encoder layers.

[0064] In one embodiment, evolutionary search can be used to search for the network architecture of a parameter-tuned hypernetwork. Evolutionary algorithms are search algorithms that simulate biological evolutionary mechanisms such as natural selection and heredity; they are essentially iterative algorithms. Evolutionary algorithms involve the following concepts: (1) Population: a subset of the solution space, corresponding to several models.

[0065] (2) Individual: is a solution in a subset, i.e. a certain model.

[0066] (3) Encoding: Describe the search object using computer language, such as representing the network structure with a fixed-length binary string.

[0067] The iterative process of evolutionary algorithm-based search involves evaluating individuals from an initial population, generating a new generation, and then evaluating individuals again until a stopping condition is met. In this search process, each iteration moves from one set of solutions to a better set, with the core issue being how to generate a new generation. Each generation requires three steps: selection, crossover, and mutation. The selection process chooses better individuals from the population, such as models with higher accuracy. The crossover process facilitates information exchange between different superior individuals, such as swapping modules between two good models. The mutation process involves making small changes to individuals; compared to the crossover process, it introduces more randomness and helps escape local optima. Using evolutionary algorithms for network search requires encoding the network into a population, iterating through the population, and translating the population back into the network.

[0068] In this invention, evolutionary search is performed on a trained supernet. For example, the accuracy of a downstream text classification task can be used as a screening criterion for a "good model" in the evolutionary iteration, and the preferred individual as the target model is obtained when the accuracy exceeds a certain threshold through iteration.

[0069] exist Figure 2 In the example, DMasking can be used to train the original PLM as a supernetwork. Then, a network architecture search (preferably an evolutionary search as described above) is performed directly on the trained supernetwork in the dimensions of network width (the number of heads in the MHA and the median size of the FFN) and network depth. The result is a performance-optimized subnetwork used as the compressed target network. In other embodiments, the search space can be pruned before performing the evolutionary search to reduce search costs. Figure 6 The compression process of a pre-trained language model according to a preferred embodiment of the present invention is illustrated. As shown in the figure, after training the hypernetwork using DMasking and before evolutionary search, an additional search space pruning step can be included. This pruning step can also be combined with DMasking; for example, by ranking the n mask paths of the objective function layer according to importance, the top k paths can be retained, thereby reducing the cost of subsequent searches.

[0070] Therefore, the pre-trained language model compression method of the present invention may further include: after the supernetwork training is completed, obtaining the importance ranking of the mask paths according to the distribution of the differentiable structural parameters of each block, and removing those paths whose importance ranking is not high. k The mask path of each, where k It is less than n The positive integer, and the network architecture search is performed on pruned supernetworks.

[0071] As mentioned earlier, in the first The MHA or FFN of each BERT layer contains Patch selection. If The search space, with its flexible width and depth, can reach a size of [missing information]. For BERT-based models, when =4, At that time, the size of the search space was approximately 3.0 × 10⁻⁶. 14 The vast search space makes subsequent evolutionary search extremely costly.

[0072] To reduce the overhead of evolutionary search, efforts can be made to narrow the search space after training the supernet. Figure 7 An example of using DMasking for search space pruning is shown, where part (a) shows the block The distribution of structural parameters is shown in section (b), which illustrates the selection of the two most important paths for subsequent evolutionary searches. Figure 7 As shown, the structural parameters of the block (MHA or FFN) in the example in part (a) can be used. ( The importance ranking of the mask paths is derived from the distribution of (=1, 2, 3, 4). Figure 7 In the example, m2 and m3 are ranked higher in importance.

[0073] because The condition is satisfied in equation (6), and The former One value is more important than the others, therefore the first one can be chosen. A discrete search is performed on each path (as shown in section (b)). Figure 7 In the example, it is shown . This means that the secondary paths in each block have the potential to achieve better compression than the top-1 paths. Since m2 and m3 are ranked higher in importance, paths 2 and 3 are retained, and paths 1 and 3 are pruned, as shown in section (b). At this point, the number of search candidates in each block decreases from... Reduced to And the size of the search space is reduced to For BERT-based models, when and At that time, the size of the search space was approximately 22.0 × 10⁻⁶. 6 By using DMasking and search space pruning, the time overhead of end-to-end architecture search can be significantly reduced.

[0074] Following evolutionary search, the compressed BERT can optionally be fine-tuned to maintain accuracy. At this point, the original pre-trained language model can be used as the teacher model to fine-tune the compressed target network based on knowledge distillation. To meet the requirements of a highly compressed BERT model, decoupled knowledge distillation (DKD) can be introduced in one embodiment. The DKD decoupling scheme uses a weighted average of target class knowledge distillation (TCKD) and non-target class knowledge distillation (NCKD) to fine-tune the compressed target network. The DKD decoupling scheme is particularly effective for multi-class cases where a balance needs to be struck between target class knowledge distillation and non-target class knowledge distillation.

[0075] Where α = 1.0 and β = 2.0. However, most NLP tasks in the GLUE benchmark correspond to 2-class or 3-class cases, therefore NCKD should be improved by incorporating more non-target knowledge than the original definition. For each mini-batch input, NCKD is redefined using a non-target probability distribution flattened along the batch dimension, and the softmax probability is calculated as follows:

[0076] in It's a class number. It refers to the batch size. This represents the predicted logit (logit is the model output vector before being fed into softmax). It is the real target.

[0077] The compression method for the pre-trained language model of the present invention has been described in detail above with reference to the accompanying drawings. Although the above description is mainly based on the BERT model, it should be understood that the present invention is applicable to other pre-trained language models, especially other pre-trained language models that include Transformer sub-model structures.

[0078] After obtaining the compressed target model according to the method of the present invention, the above model can be deployed on a mobile device. Figure 8 A compressed model system according to an embodiment of the present invention is illustrated. The system includes a server 810, a database 820, and a client (which may be a mobile device) 830. The server 810 can obtain a pre-trained language model, such as the structure and pre-training parameters of BERT, from the database 820, compress the BERT model according to the method of the present invention, and send the compressed BERT model to the mobile device 830. The mobile device 830 can then use the locally deployed model to complete interaction with the user. Therefore, the present invention can also be implemented as a mobile device deployment method for a pre-trained language model, including: deploying the target network obtained above on the mobile device; and the mobile device using the target network to perform downstream natural language processing tasks. Additionally, although... Figure 8 The image shows a mobile device 830 as a laptop computer, but the compression model obtained according to the present invention can be deployed on various types of mobile devices, such as mobile phones, smart cars, etc.

[0079] Figure 9 A schematic diagram of a computing device that can be used to implement the above-described pre-trained language model compression method according to an embodiment of the present invention is shown.

[0080] See Figure 9 The computing device 900 includes a memory 910 and a processor 920.

[0081] Processor 920 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 920 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, processor 920 may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0082] Memory 910 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 920 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 910 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some implementations, memory 910 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital versatile optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, minSD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0083] The memory 910 stores executable code, which, when processed by the processor 920, enables the processor 920 to execute the pre-trained language model compression method described above.

[0084] The pre-trained language model compression method according to the present invention has been described in detail above with reference to the accompanying drawings. Specifically, the present invention discloses a resilient BERT compression scheme with differentiable masks. The computationally intensive nature of BERT-like models hinders their deployment on resource-constrained platforms. The present invention proposes a resilient single-pass neural architecture search (NAS) framework for BERT compression, using differentiable masks, search space pruning, and evolutionary search to obtain compressed models. Specifically, Figure 6 A preferred elastic single-pass NAS framework for BERT compression is disclosed. For example... Figure 6 This paper presents an improved elastic single-pass NAS for BERT compression, achieving efficient compression with low search cost. The framework consists of supernet training with DMasking, search space pruning, and evolutionary search. In a preferred embodiment, the framework includes: • Based on the BERT model, a flexible structure search is performed in a three-dimensional search space, including the number of heads in MHA (Multi-Head Attention), the median size of FFN (Feedforward Network), and the network depth. • A pre-trained BERT model is used as a supernetwork and trained for downstream tasks. To avoid overfitting, a differentiable mask with adaptive dropout is introduced (DMasking) to enhance model training with only a few epochs.

[0085] • After training the supernet, the search space is pruned based on the DMasking structure parameters to ensure the efficiency of subsequent evolutionary search and meet resource constraints.

[0086] • Following evolutionary search, decoupled knowledge distillation (DKD) is introduced and improved to preserve the discriminative power of the highly compressed model during subnetwork fine-tuning.

[0087] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0088] Furthermore, the method according to the present invention can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing the steps defined in the above-described method of the present invention.

[0089] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) thereon, which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the various steps of the method described above according to the present invention.

[0090] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0092] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for compressing a pre-trained language model, comprising: The original pre-trained language model is used as the target model in the hypernetwork, and different widths are selected from the target function layer of the hypernetwork. n Each block is used to construct a set of subnetworks, where... n It is an integer not less than 2. For the same objective function layer, the wider block covers all the output elements contained in the narrower block. The weights of any two blocks in the objective function layer are left-aligned. The input of the objective model is an image or text. A mask layer is placed after the objective function layer of the hypernetwork, wherein the mask layer includes... n Differentiable mask paths of different widths are used to mask the output of the objective function layer with different widths, wherein the masks... n Each different width corresponds to one of the subnetwork sets. n Blocks of different widths; Determine the objective function layer for n Aggregated output of different mask paths; Based on the aggregated output, the supernetwork is trained according to a predetermined penalty function to adjust its parameters. The importance ranking of the mask paths is obtained according to the distribution of the differentiable structural parameters of each block, and paths that are not ranked first in importance are pruned. k The mask path of each, where k It is less than n positive integers, and Network architecture search is performed on pruned hypernetworks; A network architecture search is performed on the trained supernetwork to determine a compressed target network with optimized performance from the set of subnetworks; During the training of the supernetwork, for each batch of training data, a predetermined drop rate adapted to the width of the mask path is assigned to different mask paths.

2. The method as described in claim 1, wherein, The original BERT consists of multiple stacked identical unit structures, each including a multi-head attention (MHA) structure and a feedforward network (FFN) structure. The objective function layer of the hypernetwork includes: The splicing layer of the MHA structure, and the mask layer placed after the splicing layer, are used to control the number of heads in the MHA structure; and / or The first linear layer of the FFN structure, and the mask layer disposed after the first linear layer, are used to control the intermediate dimensions of the FFN structure.

3. The method as described in claim 1, wherein, Network architecture search for trained supernetworks includes performing resilient structure search in a search space of at least the following dimensions: Number of heads in the MHA structure; The intermediate dimensions of the FFN structure; and Network depth.

4. The method of claim 1, further comprising: The compressed target network is fine-tuned based on knowledge distillation using the original pre-trained language model as the teacher model.

5. The method of claim 4, wherein, Using the original pre-trained language model as the teacher model, the fine-tuning of the compressed target network based on knowledge distillation includes: The compressed target network is fine-tuned using a weighted average of target-class knowledge distillation and non-target-class knowledge distillation.

6. The method of claim 1, wherein, Evolutionary search is used to search for the network architecture of a trained supernetwork. The accuracy of downstream tasks is used as the selection criterion for individual models in the evolutionary iteration. The iteration stops when the accuracy reaches a predetermined threshold and the individual model with the highest accuracy is selected as the target model.

7. A method for deploying a pre-trained language model on a mobile device, comprising: Deploy the target network obtained according to any one of claims 1-6 on a mobile device; as well as The mobile device uses the target network to perform downstream natural language processing tasks.

8. A computing device, comprising: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

9. A computer program product comprising executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-7.

10. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-7.

Images