Pre-trained model structured pruning method and system for module clustering and centroid selection
By employing a pre-trained model structured pruning method based on module clustering and centroid selection, the problems of resource waste and performance loss in pre-trained language model compression are solved, achieving efficient model compression and performance preservation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2024-05-17
- Publication Date
- 2026-06-09
AI Technical Summary
Existing pre-trained language models suffer from resource waste and performance loss during compression, especially due to the energy and time consumption caused by multiple knowledge distillations and retraining during pruning.
A pre-trained model structured pruning method using module clustering and centroid selection is adopted. By calculating the cosine similarity and clustering between modules, redundant modules are removed and the best-performing modules are retained, thus realizing a framework of pruning first and then fine-tuning.
It effectively reduces the time and resource consumption for fine-tuning, accurately measures the redundancy of model modules, preserves model performance to the greatest extent, and achieves efficient model compression.
Smart Images
Figure CN118468965B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing neural network model compression, specifically to a structured pruning method and system for pre-trained models with module clustering and centroid selection. Background Technology
[0002] Pre-trained language models are language models pre-trained in a self-supervised manner on large-scale corpora. By pre-training on large-scale text corpora, pre-trained language models can learn rich language representations and structures, enabling them to perform well on various natural language processing tasks. However, despite their remarkable performance, the large number of parameters and computational resource requirements of pre-trained language models have become major obstacles to their practical application. As the use of pre-trained language models in various application scenarios becomes more widespread, the need to compress and accelerate them is becoming increasingly urgent. Pruning techniques are one effective compression method, reducing model size by removing unnecessary parameters and connections, thereby reducing computational overhead. Structured pruning is a common technique in this field, as it preserves the overall structure of the model, making it easier to accelerate on hardware. Therefore, how to measure the importance of different modules of a pre-trained language model and prune them to maximize model performance is an important and challenging task.
[0003] Chinese invention patent publication number CN117852593A, application number CN202311737407.8, discloses a compression method using distillation-perception mixed precision quantization. This method utilizes quantization compression and knowledge distillation to reduce the number of model parameters while preserving model performance. The method includes: determining the parameters of each neural network layer, determining the quantization bit depth of the corresponding neural network layer, quantizing the neural network model, performing multiple iterations of knowledge distillation, and selecting the neural network model with the highest accuracy after knowledge distillation as the final quantized and compressed neural network model. This patent uses a method of quantization followed by knowledge distillation for model compression and requires multiple knowledge distillations, without considering the energy and time consumption of retraining, which leads to serious resource waste. Summary of the Invention
[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a structured pruning method and system for pre-trained models of module clustering and centroid selection.
[0005] A structured pruning method for pre-trained models of module clustering and centroid selection provided by the present invention includes:
[0006] Step S1: Use the pre-trained language model and downstream task dataset to obtain the latent representation of each layer of the model;
[0007] Step S2: Calculate the size constraints for different modules based on the given model size constraints;
[0008] Step S3: Calculate the cosine similarity between different modules using the hidden representation of each layer;
[0009] Step S4: Cluster different modules based on the calculated cosine similarity matrix;
[0010] Step S5: Determine the number of categories using the clustering tree and size constraints of different modules;
[0011] Step S6: Calculate the average magnitude of different modules based on the hidden representation of each layer, retain the module with the largest magnitude in each category cluster, and remove the other modules in the category cluster.
[0012] Preferably, in step S1:
[0013] Several data samples are randomly sampled from the downstream task dataset and input into the pre-trained language model to obtain the input latent representation of the attention layer output matrix and the input latent representation of the projection matrix under the feedforward network in each Transformer layer. For each Transformer layer, there are input representations. Where T represents the length of the input sequence and d represents the dimension of the embedded latent representation; It is a vector space;
[0014] The output of the attention layer or feedforward network in the Transformer layer is written as LayerNorm(X+Sub(X)), where Sub(X) represents the function of the attention layer or feedforward network, i.e., MHA(X) or FFN(X). The function of the attention layer is written as:
[0015]
[0016] in, It is a bias. It is the hidden representation of the i-th attention head. It is the output matrix of the i-th attention head, where A represents the attention head; the dimension d of each attention head is... h The dimensional relationship between the embedded latent representation d and the dimensionality of the representation d satisfies
[0017] Hidden representations of feedforward networks in The weight matrix represents the output matrix of the feedforward network. Represents bias, d f σ represents the number of channels, σ(.) represents the activation function, such as GELU in the BERT model, and F represents the feedforward network;
[0018] The function of the feedforward network layer is written as:
[0019]
[0020] in, The feedforward network hidden representation represents the j-th channel. This represents the weight value of the j-th channel in the output matrix of the feedforward network. This represents the bias of the feedforward network output matrix;
[0021] By inputting downstream task data into the pre-trained language model, the hidden representations of each attention layer and feedforward network layer are obtained. For layer (l), the following is obtained: and
[0022] Preferably, in step S2:
[0023] The original model size is I, and the original attention layer and feedforward network layer sizes are respectively... and The size of the pruned model is limited to C, and the sizes of the attention layer and feedforward network layer after pruning are respectively... and
[0024] The above variables are related as follows: Since the attention layer and the feedforward network layer have different sensitivities to pruning, the pruning scale of the two networks is balanced using the hyperparameter λ. Their relationship is expressed by the following formula:
[0025]
[0026] Calculate the size constraints for the attention layer and the feedforward network layer using the given model size constraints.
[0027] Preferably, in step S3:
[0028] The cosine similarity between different modules is calculated using the hidden representation of each layer. For the feedforward network layer, the similarity between the hidden representations of different input channels is calculated. The similarity between the i-th and j-th channels of the (l)-th layer is obtained by calculating the hidden representations of the two channels. and The cosine similarity is obtained and expressed as the following formula:
[0029]
[0030] Where i and j take values in the range i,j∈{1,…,d} f};
[0031] Using the above formula, we obtain the similarity matrix between different channels of the feedforward network layer (l). Each element in this similarity matrix takes a value between 0 and 1, and the closer the value is to 1, the more similar the corresponding channels are, and therefore the more redundant it is. Using this similarity matrix, the distance matrix is obtained:
[0032]
[0033] Where J is an all-one matrix, the smaller the value of the element in the two distance matrices, the closer the corresponding two channels are, and therefore more redundant;
[0034] For each attention layer, the similarity between the hidden representations of different attention heads is calculated; unlike the feedforward network layer where each channel has only one dimension, each attention head in the attention layer has d dimensions. h The matching matrix is obtained by matching each dimension of different attention heads in the following ways:
[0035]
[0036] in Let be the similarity between the i-th dimension of the m-th attention head and the j-th dimension of the n-th attention head. This similarity is calculated by taking the cosine similarity of the latent representations corresponding to the two dimensions. The values of the above variables are in the range of: i,j∈{1,...,H}, m,n∈{1,...,d}. h Using this matching matrix, first match the two dimensions with the highest similarity between the two attention heads, and then match the other dimensions in descending order of similarity.
[0037] After dimensional matching, the average of the matched dimensional similarities is used as the overall similarity between the two attention heads. The overall similarity between the i-th and j-th attention heads is... Represented as:
[0038]
[0039] Here, Map(.) is a matching function that processes the matching matrix. The similarity in the matrix is matched in descending order according to the dimensions of the two attention heads, and an H-dimensional similarity vector is returned, resulting in a similarity matrix between different attention heads.
[0040] The similarity matrix is transformed into a distance matrix, a process represented by the following formula:
[0041]
[0042] Preferably, in step S4:
[0043] Obtain the distance matrix and Clustering is performed on different attention heads in the attention layer and different channels in the feedforward network layer, respectively, and hierarchical clustering is used to... and Clustering is performed to obtain a clustering tree, which is clustered according to the distance between different channels or attention heads from small to large, and different clustering results are obtained by applying different thresholds.
[0044] Preferably, in step S5:
[0045] For the feedforward network layer, the clustering threshold of the obtained clustering tree is gradually decreased from 1 at preset intervals until the number of clusters meets the size constraints for the attention layer and the feedforward network layer. For the (l)th layer, we get There are k categories of clusters, and the i-th category of clusters has k categories. i One channel, among which
[0046] For the attention layer, the clustering threshold of the obtained clustering tree is gradually decreased from 1 at preset intervals until the number of clusters meets the size constraints for the attention layer and the feedforward network layer. For the (l)th layer, we get There are k categories of clusters, and the i-th category of clusters has k categories. i One channel, among which
[0047] Preferably, in step S6:
[0048] Using the obtained hidden representations of each layer, the average magnitude of the hidden representation of each channel in the feedforward network layer and the hidden representation of each attention head in the attention layer is calculated. Then, the obtained... and There are several category clusters, and the hidden representation with the largest amplitude in each category cluster is retained, while channels or attention heads in the category clusters are pruned.
[0049] A structured pruning system for pre-trained models of module clustering and centroid selection, provided by the present invention, includes:
[0050] Module M1: Uses a pre-trained language model and downstream task datasets to obtain the latent representations of each layer of the model;
[0051] Module M2: Calculates the size constraints for different modules based on the given model size constraints;
[0052] Module M3: Calculate the cosine similarity between different modules using the hidden representations of each layer;
[0053] Module M4: Clusters different modules based on the calculated cosine similarity matrix;
[0054] Module M5: Determines the number of categories using clustering trees and size constraints for different modules;
[0055] Module M6: Calculates the average magnitude of different modules based on the latent representation of each layer, retains the module with the largest magnitude in each category cluster, and removes other modules in the category cluster.
[0056] Preferably, in module M1:
[0057] Several data samples are randomly sampled from the downstream task dataset and input into the pre-trained language model to obtain the input latent representation of the attention layer output matrix and the input latent representation of the projection matrix under the feedforward network in each Transformer layer. For each Transformer layer, there are input representations. Where T represents the length of the input sequence and d represents the dimension of the embedded latent representation; It is a vector space;
[0058] The output of the attention layer or feedforward network in the Transformer layer is written as LayerNorm(X+Sub(X)), where Sub(X) represents the function of the attention layer or feedforward network, i.e., MHA(X) or FFN(X). The function of the attention layer is written as:
[0059]
[0060] in, It is a bias. It is the hidden representation of the i-th attention head. It is the output matrix of the i-th attention head, where A represents the attention head; the dimension d of each attention head is... h The dimensional relationship between the embedded latent representation d and the dimensionality of the representation d satisfies
[0061] Hidden representations of feedforward networks in The weight matrix represents the output matrix of the feedforward network. Represents bias, d f σ represents the number of channels, σ(.) represents the activation function, such as GELU in the BERT model, and F represents the feedforward network;
[0062] The function of the feedforward network layer is written as:
[0063]
[0064] in, The feedforward network hidden representation represents the j-th channel. This represents the weight value of the j-th channel in the output matrix of the feedforward network. This represents the bias of the feedforward network output matrix;
[0065] By inputting downstream task data into the pre-trained language model, the hidden representations of each attention layer and feedforward network layer are obtained. For layer (l), the following is obtained: and
[0066] In module M2:
[0067] The original model size is I, and the original attention layer and feedforward network layer sizes are respectively... and The size of the pruned model is limited to C, and the sizes of the attention layer and feedforward network layer after pruning are respectively... and
[0068] The above variables are related as follows: Since the attention layer and the feedforward network layer have different sensitivities to pruning, the pruning scale of the two networks is balanced using the hyperparameter λ. Their relationship is expressed by the following formula:
[0069]
[0070] Calculate the size constraints for the attention layer and the feedforward network layer using the given model size constraints;
[0071] In module M3:
[0072] The cosine similarity between different modules is calculated using the hidden representation of each layer. For the feedforward network layer, the similarity between the hidden representations of different input channels is calculated. The similarity between the i-th and j-th channels of the (l)-th layer is obtained by calculating the hidden representations of the two channels. and The cosine similarity is obtained and expressed as the following formula:
[0073]
[0074] Where i and j take values in the range i,j∈{1,…,d} f};
[0075] Using the above formula, we obtain the similarity matrix between different channels of the feedforward network layer (l). Each element in this similarity matrix takes a value between 0 and 1, and the closer the value is to 1, the more similar the corresponding channels are, and therefore the more redundant it is. Using this similarity matrix, the distance matrix is obtained:
[0076]
[0077] Where J is an all-one matrix, the smaller the value of the element in the two distance matrices, the closer the corresponding two channels are, and therefore more redundant;
[0078] For each attention layer, the similarity between the hidden representations of different attention heads is calculated; unlike the feedforward network layer where each channel has only one dimension, each attention head in the attention layer has d dimensions. h The matching matrix is obtained by matching each dimension of different attention heads in the following ways:
[0079]
[0080] in Let be the similarity between the i-th dimension of the m-th attention head and the j-th dimension of the n-th attention head. This similarity is calculated by taking the cosine similarity of the latent representations corresponding to the two dimensions. The values of the above variables are in the range of: i,j∈{1,...,H}, m,n∈{1,...,d}. h Using this matching matrix, first match the two dimensions with the highest similarity between the two attention heads, and then match the other dimensions in descending order of similarity.
[0081] After dimensional matching, the average of the matched dimensional similarities is used as the overall similarity between the two attention heads. The overall similarity between the i-th and j-th attention heads is... Represented as:
[0082]
[0083] Here, Map(.) is a matching function that processes the matching matrix. The similarity in the matrix is matched in descending order according to the dimensions of the two attention heads, and an H-dimensional similarity vector is returned, resulting in a similarity matrix between different attention heads.
[0084] The similarity matrix is transformed into a distance matrix, a process represented by the following formula:
[0085]
[0086] Preferably, in module M4:
[0087] Obtain the distance matrix and Clustering is performed on different attention heads in the attention layer and different channels in the feedforward network layer, respectively, and hierarchical clustering is used to... and Clustering is performed to obtain a clustering tree, which is clustered according to the distance between different channels or attention heads from small to large, and different clustering results are obtained by applying different thresholds.
[0088] In module M5:
[0089] For the feedforward network layer, the clustering threshold of the obtained clustering tree is gradually decreased from 1 at preset intervals until the number of clusters meets the size constraints for the attention layer and the feedforward network layer. For the (l)th layer, we get There are k categories of clusters, and the i-th category of clusters has k categories. i One channel, among which
[0090] For the attention layer, the clustering threshold of the obtained clustering tree is gradually decreased from 1 at preset intervals until the number of clusters meets the size constraints for the attention layer and the feedforward network layer. For the (l)th layer, we get There are k categories of clusters, and the i-th category of clusters has k categories. i One channel, among which
[0091] In module M6:
[0092] Using the obtained hidden representations of each layer, the average magnitude of the hidden representation of each channel in the feedforward network layer and the hidden representation of each attention head in the attention layer is calculated. Then, the obtained... and There are several category clusters, and the hidden representation with the largest amplitude in each category cluster is retained, while channels or attention heads in the category clusters are pruned.
[0093] Compared with the prior art, the present invention has the following beneficial effects:
[0094] 1. This invention proposes a pre-pruning framework. Compared with the traditional task-oriented compression framework that fine-tunes before pruning, the pre-pruning framework proposed in this invention implements a framework that prunes before fine-tuning, which effectively saves time and resources during fine-tuning and eliminates the need for retraining.
[0095] 2. This invention proposes a structured pruning method based on module clustering and centroid selection for the proposed pre-pruning framework. This method can efficiently measure the similarity of different modules in a language pre-training model.
[0096] 3. This invention utilizes the proposed pre-pruning framework and structured pruning method to accurately and effectively measure the redundancy of different modules and perform targeted pruning, thereby preserving the performance of the original model to the greatest extent. Attached Figure Description
[0097] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0098] Fig. 1 This is a flowchart of the pre-pruning framework in an embodiment of the present invention;
[0099] Fig. 2 This is a flowchart of the pruning method based on module clustering and centroid selection in an embodiment of the present invention;
[0100] Fig. 3 This is a schematic diagram of the pruning method based on module clustering and centroid selection in an embodiment of the present invention. Detailed Implementation
[0101] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.
[0102] Example 1:
[0103] This invention relates to the field of language pre-trained model compression for specific tasks. This invention provides a structured pruning technique for pre-trained models based on module clustering and centroid selection. This invention utilizes the characteristic that redundant modules in pre-training often exhibit similar patterns, proposes a structured pruning method based on module clustering and centroid selection, and proposes a pruning norm for pre-pruning. It achieves high language task performance with the same model size, while also reducing the time required for fine-tuning.
[0104] According to the present invention, a structured pruning technique for pre-trained language models based on module clustering and centroid selection is provided, such as... Figs. 1-3 As shown, it includes:
[0105] Step S1: Obtain the latent representation of each layer of the model using the pre-trained language model and the downstream task dataset;
[0106] Step S2: Calculate the size constraints for different modules using the given model size constraints;
[0107] Step S3: Calculate the cosine similarity between different modules using the hidden representation of each layer;
[0108] Step S4: Cluster different modules using the calculated distance matrix;
[0109] Step S5: Determine the number of categories using the clustering tree and the size constraints of different modules;
[0110] Step S6: Calculate the average magnitude of different modules using the hidden representation of each layer, retain the module with the largest magnitude in each category cluster, and remove the other modules in the category cluster.
[0111] First, several data samples are randomly sampled from the downstream task dataset and input into the pre-trained language model to obtain the input latent representation of the attention layer output matrix and the input latent representation of the projection matrix under the feedforward network in each Transformer layer. Specifically, for each Transformer layer, there are input representations... Where T represents the length of the input sequence and d represents the dimension of the embedded latent representation, the output of the attention layer or feedforward network in the Transformer layer can be written as LayerNorm(X+Sub(X)), where Sub(X) represents the function of the attention layer or feedforward network, i.e., MHA(X) or FFN(X). Specifically, the function of the attention layer can be written as:
[0112]
[0113] in, It is a bias. It is the hidden representation of the i-th attention head. It is the output matrix of the i-th attention head. The dimension d_h of each attention head and the dimension of the embedded latent representation d are related by the condition d_h = d / H.
[0114] Similarly, the hidden representation of a feedforward network can be defined. Where W∈R N×d The weight matrix representing the output matrix of the feedforward network, b∈R d Let σ represent the bias, N represent the number of channels, and σ(.) represent the activation function, such as GELU in the BERT model. Then the function of the feedforward network layer can be written as:
[0115]
[0116] in, The feedforward network hidden representation represents the j-th channel. This represents the weight value of the j-th channel in the output matrix of the feedforward network. This represents the bias of the output matrix of the feedforward network.
[0117] Using the above formulas, the hidden representations of each attention layer and feedforward network layer can be obtained by inputting downstream task data into the pre-trained language model. For example, for layer (l), we can obtain... and
[0118] Next, based on the given size constraints of the pruned model, the size constraints for the attention layer and the feedforward network layer are calculated separately. Given the original model size as I, the sizes of the original attention layer and the feedforward network layer are respectively... and The size of the pruned model is limited to C, and the sizes of the attention layer and feedforward network layer after pruning are respectively... and The above variables are related as follows: Since the attention layer and the feedforward network layer have different sensitivities to pruning, it is necessary to use the hyperparameter λ to balance the pruning scale of the two networks. Their relationship can be expressed by the following formula:
[0119]
[0120] Using the above formulas, the size constraints for the attention layer and the feedforward network layer can be calculated separately using the given model size constraints.
[0121] Next, the cosine similarity between different modules is calculated using the hidden representations of each layer. Specifically, for the feedforward network layer, the similarity between the hidden representations of different input channels is calculated. For example, the similarity between the i-th and j-th channels of layer (l) can be calculated by retrieving the hidden representations of the two channels. and The cosine similarity is obtained and expressed as the following formula:
[0122]
[0123] Where i and j take values in the range i,j∈{1,…,d} f Using the above formula, the similarity matrix between different channels of the feedforward network layer (l) can be obtained. Each element in this similarity matrix takes a value between 0 and 1, and the closer the value is to 1, the more similar the corresponding channels are, thus increasing redundancy and facilitating subsequent pruning. Furthermore, using this similarity matrix, the distance matrix can be obtained:
[0124]
[0125] Here, J is an all-one matrix. Correspondingly, the smaller the value of the element in the two distance matrices, the closer the corresponding two channels are, and therefore the more redundant they are.
[0126] For each attention layer, the similarity between the hidden representations of different attention heads is calculated. Unlike feedforward networks where each channel has only one dimension, each attention head in an attention layer has d dimensions. hThere are several dimensions. Therefore, to measure the similarity between different attention heads, we need to first match each dimension of the different attention heads, and obtain the following matching matrix:
[0127]
[0128] in Let be the similarity between the i-th dimension of the m-th attention head and the j-th dimension of the n-th attention head. Similarly, this similarity is calculated by evaluating the cosine similarity of the latent representations corresponding to the two dimensions. The values of the above variables range from: i,j∈{1,...,H}, m,n∈{1,...,d}. h Using this matching matrix, we first match the two dimensions with the highest similarity between the two attention heads, and then match other dimensions in a similar manner from high to low similarity.
[0129] After dimensional matching, the average of the matched dimensional similarities across the dimensions is used as the overall similarity between the two attention heads. Taking the i-th and j-th attention heads as an example, their overall similarity is... It can be represented as:
[0130]
[0131] Here, Map(.) is a matching function that processes the matching matrix. The similarity in the matrix is matched in descending order according to the dimensions of the two attention heads, and an H-dimensional similarity vector is returned. Finally, a similarity matrix between different attention heads is obtained.
[0132] Similarly, the similarity matrix can be transformed into a distance matrix, a process that can be represented by the following formula:
[0133]
[0134] Using the distance matrix calculated above and Clustering is performed on different attention heads in the attention layer and different channels in the feedforward network layer, respectively. Specifically, hierarchical clustering is used to... and Clustering is performed to obtain a clustering tree, which is clustered according to the distance between different channels or attention heads from small to large, and different clustering results can be obtained by using different thresholds in the future.
[0135] For the feedforward network layer, using the clustering tree obtained above, the clustering threshold starts from 1 and decreases sequentially at certain intervals until the number of clusters meets the size constraints of the attention layer and the feedforward network layer obtained in step S2. Finally, for layer (l), we obtain... There are k categories of clusters, and the i-th category of clusters has k categories. i One channel, among which
[0136] Similarly, for the attention layer, using the clustering tree obtained above, the clustering threshold starts from 1 and decreases successively at certain intervals until the number of clusters obtained meets the size constraints for the attention layer and the feedforward network layer obtained in step S2. Finally, for layer (l), we obtain... There are k categories of clusters, and the i-th category of clusters has k categories. i One channel, among which
[0137]
[0138] Finally, using the hidden representations of each layer obtained in step S1, the average magnitude of the hidden representations of each channel in the feedforward network layer and the hidden representations of each attention head in the attention layer is calculated. Then, the value obtained in step S5 is used... and There are several category clusters, and the hidden representation with the largest amplitude in each category cluster is retained, while channels or attention heads in the category clusters are pruned.
[0139] Example 2:
[0140] Example 2 is a preferred embodiment of Example 1, and is used to illustrate the present invention in more detail.
[0141] This invention also provides a pre-trained model structured pruning system for module clustering and centroid selection. The pre-trained model structured pruning system for module clustering and centroid selection can be implemented by executing the process steps of the pre-trained model structured pruning method for module clustering and centroid selection. That is, those skilled in the art can understand the pre-trained model structured pruning method for module clustering and centroid selection as a preferred embodiment of the pre-trained model structured pruning system for module clustering and centroid selection.
[0142] A structured pruning system for pre-trained models of module clustering and centroid selection, provided by the present invention, includes:
[0143] Module M1: Uses a pre-trained language model and downstream task datasets to obtain the latent representations of each layer of the model;
[0144] Preferably, in module M1:
[0145] Several data samples are randomly sampled from the downstream task dataset and input into the pre-trained language model to obtain the input latent representation of the attention layer output matrix and the input latent representation of the projection matrix under the feedforward network in each Transformer layer. For each Transformer layer, there are input representations. Where T represents the length of the input sequence and d represents the dimension of the embedded latent representation; It is a vector space;
[0146] The output of the attention layer or feedforward network in the Transformer layer is written as LayerNorm(X+Sub(X)), where Sub(X) represents the function of the attention layer or feedforward network, i.e., MHA(X) or FFN(X). The function of the attention layer is written as:
[0147]
[0148] in, It is a bias. It is the hidden representation of the i-th attention head. It is the output matrix of the i-th attention head, where A represents the attention head; the dimension d of each attention head is... h The dimensional relationship between the embedded latent representation d and the dimensionality of the representation d satisfies
[0149] Hidden representations of feedforward networks in The weight matrix represents the output matrix of the feedforward network. Represents bias, d f σ represents the number of channels, σ(.) represents the activation function, such as GELU in the BERT model, and F represents the feedforward network;
[0150] The function of the feedforward network layer is written as:
[0151]
[0152] in, The feedforward network hidden representation represents the j-th channel. This represents the weight value of the j-th channel in the output matrix of the feedforward network. This represents the bias of the feedforward network output matrix;
[0153] By inputting downstream task data into the pre-trained language model, the hidden representations of each attention layer and feedforward network layer are obtained. For layer (l), the following is obtained:
[0154] Module M2: Calculates the size constraints for different modules based on the given model size constraints;
[0155] In module M2:
[0156] The original model size is I, and the original attention layer and feedforward network layer sizes are respectively... and The size of the pruned model is limited to C, and the sizes of the attention layer and feedforward network layer after pruning are respectively... and
[0157] The above variables are related as follows: Since the attention layer and the feedforward network layer have different sensitivities to pruning, the pruning scale of the two networks is balanced using the hyperparameter λ. Their relationship is expressed by the following formula:
[0158]
[0159] Calculate the size constraints for the attention layer and the feedforward network layer using the given model size constraints;
[0160] Module M3: Calculate the cosine similarity between different modules using the hidden representations of each layer;
[0161] In module M3:
[0162] The cosine similarity between different modules is calculated using the hidden representation of each layer. For the feedforward network layer, the similarity between the hidden representations of different input channels is calculated. The similarity between the i-th and j-th channels of the (l)-th layer is obtained by calculating the hidden representations of the two channels. and The cosine similarity is obtained and expressed as the following formula:
[0163]
[0164] Where i and j take values in the range i,j∈{1,…,d} f};
[0165] Using the above formula, we obtain the similarity matrix between different channels of the feedforward network layer (l). Each element in this similarity matrix takes a value between 0 and 1, and the closer the value is to 1, the more similar the corresponding channels are, and therefore the more redundant it is. Using this similarity matrix, the distance matrix is obtained:
[0166]
[0167] Where J is an all-one matrix, the smaller the value of the element in the two distance matrices, the closer the corresponding two channels are, and therefore more redundant;
[0168] For each attention layer, the similarity between the hidden representations of different attention heads is calculated; unlike the feedforward network layer where each channel has only one dimension, each attention head in the attention layer has d dimensions. h The matching matrix is obtained by matching each dimension of different attention heads in the following ways:
[0169]
[0170] in Let be the similarity between the i-th dimension of the m-th attention head and the j-th dimension of the n-th attention head. This similarity is calculated by taking the cosine similarity of the latent representations corresponding to the two dimensions. The values of the above variables are in the range of: i,j∈{1,...,H}, m,n∈{1,...,d}. h Using this matching matrix, first match the two dimensions with the highest similarity between the two attention heads, and then match the other dimensions in descending order of similarity.
[0171] After dimensional matching, the average of the matched dimensional similarities is used as the overall similarity between the two attention heads. The overall similarity between the i-th and j-th attention heads is... Represented as:
[0172]
[0173] Here, Map(.) is a matching function that processes the matching matrix. The similarity in the matrix is matched in descending order according to the dimensions of the two attention heads, and an H-dimensional similarity vector is returned, resulting in a similarity matrix between different attention heads.
[0174] The similarity matrix is transformed into a distance matrix, a process represented by the following formula:
[0175]
[0176] Module M4: Clusters different modules based on the calculated cosine similarity matrix;
[0177] Preferably, in module M4:
[0178] Obtain the distance matrix and Clustering is performed on different attention heads in the attention layer and different channels in the feedforward network layer, respectively, and hierarchical clustering is used to... and Clustering is performed to obtain a clustering tree, which is clustered according to the distance between different channels or attention heads from small to large, and different clustering results are obtained by applying different thresholds.
[0179] Module M5: Determines the number of categories using clustering trees and size constraints for different modules;
[0180] In module M5:
[0181] For the feedforward network layer, the clustering threshold of the obtained clustering tree is gradually decreased from 1 at preset intervals until the number of clusters meets the size constraints for the attention layer and the feedforward network layer. For the (l)th layer, we get There are k categories of clusters, and the i-th category of clusters has k categories. i One channel, among which
[0182] For the attention layer, the clustering threshold of the obtained clustering tree is gradually decreased from 1 at preset intervals until the number of clusters meets the size constraints for the attention layer and the feedforward network layer. For the (l)th layer, we get There are k categories of clusters, and the i-th category of clusters has k categories. i One channel, among which
[0183] Module M6: Calculates the average magnitude of different modules based on the latent representation of each layer, retains the module with the largest magnitude in each category cluster, and removes other modules in the category cluster.
[0184] In module M6:
[0185] Using the obtained hidden representations of each layer, the average magnitude of the hidden representation of each channel in the feedforward network layer and the hidden representation of each attention head in the attention layer is calculated. Then, the obtained... and There are several category clusters, and the hidden representation with the largest amplitude in each category cluster is retained, while channels or attention heads in the category clusters are pruned.
[0186] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0187] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. A structured pruning method for pre-trained models of module clustering and centroid selection, characterized in that, include: Step S1: Use the pre-trained language model and downstream task dataset to obtain the latent representation of each layer of the model; Step S2: Calculate the size constraints for different modules based on the given model size constraints; Step S3: Calculate the cosine similarity between different modules using the hidden representation of each layer; Step S4: Cluster different modules based on the calculated cosine similarity matrix; Step S5: Determine the number of categories using the clustering tree and size constraints of different modules; Step S6: Calculate the average magnitude of different modules based on the latent representation of each layer, and retain the module with the largest magnitude in each category cluster, while removing other modules in the category cluster; In step S3: The cosine similarity between different modules is calculated using the hidden representations of each layer. For the feedforward network layer, the similarity between the hidden representations of different input channels is calculated. The first layer The and the first The similarity between channels is calculated by examining the latent representations of the two channels. and The cosine similarity is obtained and expressed as the following formula: in, and The range of values is ; Using the above formula, we obtain the first... Similarity matrix between different channels of the feedforward network layer In this similarity matrix, each element takes a value between 0 and 1, and the closer the element's value is to 1, the more similar the corresponding two channels are, and therefore the more redundant it is. Using this similarity matrix, the distance matrix is obtained: in, As a matrix of all ones, the smaller the value of the element in the two distance matrices, the closer the corresponding two channels are, and therefore more redundant; For each attention layer, the similarity between the hidden representations of different attention heads is calculated; unlike feedforward networks where each channel has only one dimension, each attention head in an attention layer has... The matching matrix is obtained by matching each dimension of different attention heads in the following ways: in For the first The first one in the attention. peacekeeping The first one in the attention. The similarity of the two dimensions is calculated by evaluating the cosine similarity of their latent representations, where: , Using this matching matrix, we first match the two dimensions with the highest similarity between two attention heads, and then match other dimensions in descending order of similarity. After dimensional matching, the average of the matched dimensional similarities across all dimensions is used as the overall similarity between the two attention heads. and the Overall similarity of individual attention points Represented as: in, It is a matching function that converts the matching matrix into a single matrix. The similarity in the two attention heads is matched in descending order of their dimensions, and a result is returned. A similarity vector of dimension is used to obtain a similarity matrix between different attention heads. ; The similarity matrix is transformed into a distance matrix, a process represented by the following formula: In step S4: Obtain the distance matrix and Clustering is performed on different attention heads in the attention layer and different channels in the feedforward network layer, respectively, and hierarchical clustering is used to... and Clustering is performed to obtain a clustering tree, which is clustered according to the distance between different channels or attention heads from small to large, and different clustering results are obtained by applying different thresholds.
2. The pre-trained model structured pruning method for module clustering and centroid selection according to claim 1, characterized in that, In step S1: Several data samples are randomly sampled from the downstream task dataset and input into the pre-trained language model to obtain the input latent representation of the attention layer output matrix and the input latent representation of the projection matrix under the feedforward network in each Transformer layer. For each Transformer layer, there are input representations. ,in Represents the length of the input sequence. Represents the embedded latent representation dimension; It is a vector space; The output of the attention layer or feedforward network in the Transformer layer is written as... ,in The functional function representing the attention layer or feedforward network, i.e. or The function of the attention layer is written as: in, It is a bias. It is the first The latent representation of an attention head It is the first The output matrix of each attention head, Dimensions of each attention head and embedded latent representation Dimensional relationship satisfies ; Hidden representations of feedforward networks ,in The weight matrix represents the output matrix of the feedforward network. Represents bias. Represents the number of channels. This represents the activation function; for example, in the BERT model, the activation function is GELU. It is a feedforward network; The function of the feedforward network layer is written as: in, Representing the Hidden representation of feedforward network for each channel. The output matrix of the feedforward network is represented by the first... The weight values of each channel, This represents the bias of the feedforward network output matrix; By inputting downstream task data into a pre-trained language model, the hidden representations of each attention layer and feedforward network layer are obtained, respectively. Layer, to obtain and .
3. The pre-trained model structured pruning method for module clustering and centroid selection according to claim 1, characterized in that, In step S2: The original model size is The sizes of the original attention layer and the feedforward network layer are respectively and The size of the pruned model is limited to [size not specified]. After pruning, the sizes of the attention layer and the feedforward network layer are respectively and ; in: , Since the attention layer and the feedforward network layer have different sensitivities to pruning, hyperparameters are used... The pruning scales of the two networks can be balanced, and their relationship can be expressed by the following formula: Calculate the size constraints for the attention layer and the feedforward network layer using the given model size constraints.
4. The pre-trained model structured pruning method for module clustering and centroid selection according to claim 1, characterized in that, In step S5: For the feedforward network layer, the clustering threshold of the obtained clustering tree is gradually decreased from 1 at preset intervals until the number of clusters meets the size constraints for the attention layer and the feedforward network layer. , for the Layer obtained The first category cluster, and the second category cluster. Each category has One channel, among which ; For the attention layer, the clustering threshold of the obtained clustering tree is gradually decreased from 1 at preset intervals until the number of clusters meets the size constraints for the attention layer and the feedforward network layer. , for the Layer obtained The first category cluster, and the second category cluster. Each category has One channel, among which .
5. The pre-trained model structured pruning method for module clustering and centroid selection according to claim 1, characterized in that, In step S6: Using the obtained hidden representations of each layer, the average magnitude of the hidden representation of each channel in the feedforward network layer and the hidden representation of each attention head in the attention layer is calculated. Then, the obtained... and There are several category clusters, and the hidden representation with the largest amplitude in each category cluster is retained, while channels or attention heads in the category clusters are pruned.
6. A structured pruning system for a pre-trained model of module clustering and centroid selection, characterized in that, include: Module M1: Uses a pre-trained language model and downstream task datasets to obtain the latent representations of each layer of the model; Module M2: Calculates the size constraints for different modules based on the given model size constraints; Module M3: Calculate the cosine similarity between different modules using the hidden representations of each layer; Module M4: Clusters different modules based on the calculated cosine similarity matrix; Module M5: Determines the number of categories using clustering trees and size constraints for different modules; Module M6: Calculate the average magnitude of different modules based on the latent representation of each layer, retain the module with the largest magnitude in each category cluster, and remove other modules in the category cluster; In module M3: The cosine similarity between different modules is calculated using the hidden representations of each layer. For the feedforward network layer, the similarity between the hidden representations of different input channels is calculated. The first layer The and the first The similarity between channels is calculated by examining the latent representations of the two channels. and The cosine similarity is obtained and expressed as the following formula: in, and The range of values is ; Using the above formula, we obtain the first... Similarity matrix between different channels of the feedforward network layer In this similarity matrix, each element takes a value between 0 and 1, and the closer the element's value is to 1, the more similar the corresponding two channels are, and therefore the more redundant it is. Using this similarity matrix, the distance matrix is obtained: in, As a matrix of all ones, the smaller the value of the element in the two distance matrices, the closer the corresponding two channels are, and therefore more redundant; For each attention layer, the similarity between the hidden representations of different attention heads is calculated; unlike feedforward networks where each channel has only one dimension, each attention head in an attention layer has... The matching matrix is obtained by matching each dimension of different attention heads in the following ways: in For the first The first one in the attention. peacekeeping The first one in the attention. The similarity of the two dimensions is calculated by evaluating the cosine similarity of their latent representations, where: , Using this matching matrix, we first match the two dimensions with the highest similarity between two attention heads, and then match other dimensions in descending order of similarity. After dimensional matching, the average of the matched dimensional similarities across all dimensions is used as the overall similarity between the two attention heads. and the Overall similarity of individual attention points Represented as: in, It is a matching function that converts the matching matrix into a single matrix. The similarity in the two attention heads is matched in descending order of their dimensions, and a result is returned. A similarity vector of dimension is used to obtain a similarity matrix between different attention heads. ; The similarity matrix is transformed into a distance matrix, a process represented by the following formula: In module M4: Obtain the distance matrix and Clustering is performed on different attention heads in the attention layer and different channels in the feedforward network layer, respectively, and hierarchical clustering is used to... and Clustering is performed to obtain a clustering tree, which is clustered according to the distance between different channels or attention heads from small to large, and different clustering results are obtained by applying different thresholds. In module M5: For the feedforward network layer, the clustering threshold of the obtained clustering tree is gradually decreased from 1 at preset intervals until the number of clusters meets the size constraints for the attention layer and the feedforward network layer. , for the Layer obtained The first category cluster, and the second category cluster. Each category has One channel, among which ; For the attention layer, the clustering threshold of the obtained clustering tree is gradually decreased from 1 at preset intervals until the number of clusters meets the size constraints for the attention layer and the feedforward network layer. , for the Layer obtained The first category cluster, and the second category cluster. Each category has One channel, among which .
7. The pre-trained model structured pruning system for module clustering and centroid selection according to claim 6, characterized in that: In module M1: Several data samples are randomly sampled from the downstream task dataset and input into the pre-trained language model to obtain the input latent representation of the attention layer output matrix and the input latent representation of the projection matrix under the feedforward network in each Transformer layer. For each Transformer layer, there are input representations. ,in Represents the length of the input sequence. Represents the embedded latent representation dimension; It is a vector space; The output of the attention layer or feedforward network in the Transformer layer is written as... ,in The functional function representing the attention layer or feedforward network, i.e. or The function of the attention layer is written as: in, It is a bias. It is the first The latent representation of an attention head It is the first The output matrix of each attention head, Dimensions of each attention head and embedded latent representation Dimensional relationship satisfies ; Hidden representations of feedforward networks ,in The weight matrix represents the output matrix of the feedforward network. Represents bias. Represents the number of channels. This represents the activation function; for example, in the BERT model, the activation function is GELU. It is a feedforward network; The function of the feedforward network layer is written as: in, Representing the Hidden representation of feedforward network for each channel. The output matrix of the feedforward network is represented by the first... The weight values of each channel, This represents the bias of the feedforward network output matrix; By inputting downstream task data into a pre-trained language model, the hidden representations of each attention layer and feedforward network layer are obtained, respectively. Layer, to obtain and ; In module M2: The original model size is The sizes of the original attention layer and the feedforward network layer are respectively and The size of the pruned model is limited to [size not specified]. After pruning, the sizes of the attention layer and the feedforward network layer are respectively and ; in: , Since the attention layer and the feedforward network layer have different sensitivities to pruning, hyperparameters are used... The pruning scales of the two networks can be balanced, and their relationship can be expressed by the following formula: Calculate the size constraints for the attention layer and the feedforward network layer using the given model size constraints.
8. The pre-trained model structured pruning system for module clustering and centroid selection according to claim 6, characterized in that: In module M6: Using the obtained hidden representations of each layer, the average magnitude of the hidden representation of each channel in the feedforward network layer and the hidden representation of each attention head in the attention layer is calculated. Then, the obtained... and There are several category clusters, and the hidden representation with the largest amplitude in each category cluster is retained, while channels or attention heads in the category clusters are pruned.