Perceptual word embedding dimension dynamic pruning method and device, equipment and storage medium

Abstract

The invention provides a perceptual word embedding dimension dynamic pruning method and device, equipment and a storage medium. The method comprises the steps of obtaining a word embedding matrix according to a word embedding layer of a pre-trained visual language model; carrying out double-index quantification based on the matrix, and generating a dimension importance score by calculating a weight norm and ablation statistics significance indexes; generating a binary pruning mask according to the binary pruning mask; according to the mapping relation between the column dimension of the mask and the word embedding matrix and the dimension of the Transform layer, carrying out structured pruning on the layer, and generating a pruned visual language model; and finally, through knowledge distillation, recovering the performance by using a dimension projection matrix to obtain a final pruning visual language model. According to the method, accurate dynamic pruning is realized, pruning blindness is overcome, layer dimensions are associated, cross-modal matching performance is guaranteed, and efficient parameter compression and reasoning acceleration are realized.

Description

Technical Field

[0001] This application belongs to the field of model pruning, and in particular relates to a method, apparatus, device and storage medium for dynamic pruning of perceptual word embedding dimension. Background Technology

[0002] Visual Language Models (VLMs) and their compression techniques are used to handle cross-modal tasks such as cross-modal retrieval, image classification, and visual question answering, and are widely applied in fields such as industrial inspection, intelligent security, and autonomous driving. VLMs are based on the Transformer architecture and achieve text-image alignment through pre-training, aiming to improve the model's versatility.

[0003] The core components of VLM include the word embedding table (with a parameter size of V×H, where V is the vocabulary size and H is the word embedding dimension), the Transformer encoder (including attention layers and linear layers, with the parameter dimension strongly bound to H, accounting for 60%-80% of the total model parameters), and the visual encoder. Traditional compression methods prune the Transformer layers through weight pruning, channel pruning, and other methods, relying on general strategies such as global weight norm ranking or gradient sensitivity analysis, aiming to reduce model parameters and inference time, thereby reducing storage overhead and accelerating computation.

[0004] However, in global weight norm ranking or gradient sensitivity analysis, because the semantic contribution of word embedding dimension is not associated and the one-to-one mapping between word embedding dimension and Transformer layer dimension is ignored, the pruning is highly blind and key dimensions are easily pruned in error. This results in a significant decrease in cross-modal semantic alignment performance, making it difficult to balance compression efficiency and performance preservation, and failing to meet the real-time requirements of edge device deployment. Summary of the Invention

[0005] The purpose of this application is to overcome the deficiencies in the prior art and provide a perceptual word embedding dimension dynamic pruning method, apparatus, device and storage medium.

[0006] This application provides a perceptual word embedding dimension dynamic pruning method, including:

[0007] Obtain the word embedding matrix based on the word embedding layer of the pre-trained visual language model;

[0008] The word embedding matrix is ​​subjected to sparse-induced fine-tuning to generate a fine-tuned word embedding matrix. The sparse-induced fine-tuning includes task-driven structured sparse training using Group Lasso regularization.

[0009] Based on the fine-tuned word embedding matrix, dual-indicator quantization is performed to generate dimensional importance scores. The dual-indicator quantization includes calculating the weight norm and the ablation statistical significance index.

[0010] Generate a binary pruning mask based on the aforementioned dimensional importance score;

[0011] Based on the mapping relationship between the binary pruning mask and the column dimension of the word embedding matrix and the Transformer layer dimension, the Transformer layer is structurally pruned to generate a pruned visual language model.

[0012] By using knowledge distillation and employing a dimensional projection matrix, the performance of the pruned visual language model is restored, resulting in the final pruned visual language model.

[0013] Optionally, based on the fine-tuned word embedding matrix, dual-index quantization is performed to generate a dimensional importance score, including:

[0014] Cluster analysis is performed on the weight norm to generate dimension clustering results, wherein the cluster analysis divides the dimensions into important clusters and redundant clusters;

[0015] The statistical significance index of ablation was tested repeatedly to generate confidence intervals for performance loss.

[0016] Based on the dimensional clustering results and the performance loss confidence interval, the statistical significance of the dimensional importance score is verified, and the dimensional importance score is adjusted based on the statistical significance.

[0017] Optionally, the word embedding matrix is ​​subjected to sparse-induced fine-tuning to generate a fine-tuned word embedding matrix, including:

[0018] In the sparse-induced fine-tuning, a stopping condition is set, which includes no continuous improvement in the performance of the validation set task and the proportion of redundant dimensions reaching a predetermined multiple of the target compression rate.

[0019] The training process of the sparse-induced fine-tuning is dynamically adjusted according to the stopping condition, in coordination with the target compression rate of the generated binary pruning mask.

[0020] Optionally, based on the mapping relationship between the binary pruning mask and the column dimensions of the word embedding matrix and the Transformer layer dimensions, the Transformer layer is structurally pruned to generate a pruned visual language model, including:

[0021] Construct a two-dimensional diagonal mask matrix corresponding to the binary pruning mask;

[0022] The two-dimensional diagonal mask matrix is ​​used to perform matrix multiplication on the query, key, and value projection matrices of the Transformer layer to achieve structured pruning.

[0023] Optionally, performance recovery of the pruned visual language model is performed using a dimensional projection matrix through knowledge distillation, including:

[0024] In attention layer distillation, a learnable projection matrix is ​​introduced for each head;

[0025] The learnable projection matrix is ​​used to perform dimensional mapping on the query, key, and value features of the teacher model to generate aligned attention features;

[0026] Based on the aligned attention features, cross-modal alignment distillation is performed, including: calculating a loss function through a temperature-softened similarity matrix.

[0027] Optionally, based on the fine-tuned word embedding matrix, dual-index quantization is performed to generate a dimensional importance score, including:

[0028] The loss weights in the knowledge distillation are adjusted based on the importance scores of the aforementioned dimensions.

[0029] The loss weights are adaptively optimized using a validation set, forming a feedback loop between the dual-index quantization and the knowledge distillation.

[0030] Optionally, the word embedding matrix is ​​subjected to sparse-induced fine-tuning to generate a fine-tuned word embedding matrix, including:

[0031] The sparse-induced fine-tuning and the knowledge distillation are jointly optimized using a multi-objective loss function;

[0032] In the joint optimization, the stopping condition of the sparse-induced fine-tuning optimizes the dimensional projection matrix learning of the knowledge distillation.

[0033] This application also provides a perceptual word embedding dimension dynamic pruning device, including:

[0034] The acquisition module obtains the word embedding matrix based on the word embedding layer of the pre-trained visual language model;

[0035] The fine-tuning module performs sparse-induced fine-tuning on the word embedding matrix to generate a fine-tuned word embedding matrix. The sparse-induced fine-tuning includes task-driven structured sparse training using Group Lasso regularization.

[0036] The scoring module performs dual-indicator quantization based on the fine-tuned word embedding matrix to generate a dimension importance score. The dual-indicator quantization includes calculating the weight norm and the ablation statistical significance index.

[0037] The masking module generates a binary pruned mask based on the dimensional importance score.

[0038] The model module performs structured pruning on the Transformer layer based on the binary pruning mask and the mapping relationship between the column dimensions of the word embedding matrix and the Transformer layer dimensions, generating a pruned visual language model.

[0039] The recovery module performs performance recovery on the pruned visual language model by using knowledge distillation and a dimensional projection matrix to obtain the final pruned visual language model.

[0040] Optionally, the scoring module performs dual-index quantization based on the fine-tuned word embedding matrix to generate a dimensional importance score, including:

[0041] Cluster analysis is performed on the weight norm to generate dimension clustering results, wherein the cluster analysis divides the dimensions into important clusters and redundant clusters;

[0042] The statistical significance index of ablation was tested repeatedly to generate confidence intervals for performance loss.

[0043] Based on the dimensional clustering results and the performance loss confidence interval, the statistical significance of the dimensional importance score is verified, and the dimensional importance score is adjusted based on the statistical significance.

[0044] Optionally, the fine-tuning module performs sparse-induced fine-tuning on the word embedding matrix to generate a fine-tuned word embedding matrix, including:

[0045] In the sparse-induced fine-tuning, a stopping condition is set, which includes no continuous improvement in the performance of the validation set task and the proportion of redundant dimensions reaching a predetermined multiple of the target compression rate.

[0046] The training process of the sparse-induced fine-tuning is dynamically adjusted according to the stopping condition, in coordination with the target compression rate of the generated binary pruning mask.

[0047] Optionally, the model module, based on the mapping relationship between the binary pruning mask and the column dimensions of the word embedding matrix and the Transformer layer dimensions, performs structured pruning on the Transformer layer to generate a pruned visual language model, including:

[0048] Construct a two-dimensional diagonal mask matrix corresponding to the binary pruning mask;

[0049] The two-dimensional diagonal mask matrix is ​​used to perform matrix multiplication on the query, key, and value projection matrices of the Transformer layer to achieve structured pruning.

[0050] Optionally, the recovery module performs performance recovery on the pruned visual language model using a dimensional projection matrix through knowledge distillation, including:

[0051] In attention layer distillation, a learnable projection matrix is ​​introduced for each head;

[0052] The learnable projection matrix is ​​used to perform dimensional mapping on the query, key, and value features of the teacher model to generate aligned attention features;

[0053] Based on the aligned attention features, cross-modal alignment distillation is performed, including: calculating a loss function through a temperature-softened similarity matrix.

[0054] Optionally, the scoring module performs dual-index quantization based on the fine-tuned word embedding matrix to generate a dimensional importance score, including:

[0055] The loss weights in the knowledge distillation are adjusted based on the importance scores of the aforementioned dimensions.

[0056] The loss weights are adaptively optimized using a validation set, forming a feedback loop between the dual-index quantization and the knowledge distillation.

[0057] Optionally, the fine-tuning module performs sparse-induced fine-tuning on the word embedding matrix to generate a fine-tuned word embedding matrix, including:

[0058] The sparse-induced fine-tuning and the knowledge distillation are jointly optimized using a multi-objective loss function;

[0059] In the joint optimization, the stopping condition of the sparse-induced fine-tuning optimizes the dimensional projection matrix learning of the knowledge distillation.

[0060] This application also provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0061] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the above-described method.

[0062] The beneficial effects of this application are:

[0063] This application provides a perceptual word embedding dimension dynamic pruning method, comprising: obtaining a word embedding matrix based on the word embedding layer of a pre-trained visual language model; performing sparsity-induced fine-tuning on the word embedding matrix to generate a fine-tuned word embedding matrix, wherein the sparsity-induced fine-tuning includes task-driven structured sparse training using Group Lasso regularization; performing dual-index quantization based on the fine-tuned word embedding matrix to generate a dimension importance score, wherein the dual-index quantization includes calculating the weight norm and the ablation statistical significance index; generating a binary pruning mask based on the dimension importance score; performing structured pruning on the Transformer layer based on the binary pruning mask and the mapping relationship between the column dimensions of the word embedding matrix and the Transformer layer dimensions to generate a pruned visual language model; and performing performance recovery on the pruned visual language model using a dimension projection matrix through knowledge distillation to obtain the final pruned visual language model. This application achieves precise dynamic pruning based on word embedding dimension importance assessment through the technical solution described in claim 1, overcoming the shortcomings of blind pruning in the prior art. By sparse-induced fine-tuning and dual-index quantization of the associated Transformer layer dimensions, it achieves efficient compression of model parameters and acceleration of inference, while ensuring cross-modal matching performance under specific tasks. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the dynamic pruning process for perceptual word embedding dimensions;

[0065] Figure 2 This is a structured schematic diagram of binary pruning mask generation;

[0066] Figure 3 This is a schematic diagram of the pruning process for the corresponding dimension of the Transformer layer;

[0067] Figure 4 This is a schematic diagram of the hierarchical distillation logic for knowledge distillation performance recovery;

[0068] Figure 5 This is a schematic diagram of a perceptual word embedding dimension dynamic pruning device. Detailed Implementation

[0069] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is to be understood that various forms of implementation of the present disclosure are intended and should not be limited to the embodiments set forth herein. Rather, the embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0070] Please refer to Figures 1-4As shown, this application provides a perceptual word embedding dimension dynamic pruning method, applied to the field of visual language tasks, to solve the problems of model parameter redundancy and inference acceleration. The method includes:

[0071] S101. Obtain the word embedding matrix based on the word embedding layer of the pre-trained visual language model.

[0072] Word embedding matrix representation is V is the size of the basic vocabulary, and H is the embedding dimension of the pre-trained words.

[0073] The word embedding matrix E serves as the input for subsequent processing, and its dimensionality determines the feature representation capability of the model on the text side.

[0074] S102. Perform sparse-induced fine-tuning on the word embedding matrix to generate a fine-tuned word embedding matrix. The sparse-induced fine-tuning includes task-driven structured sparse training using Group Lasso regularization.

[0075] The core objective of sparsity-induced fine-tuning is to enable the dimensions in the word embedding layer that are useless to downstream tasks to actively exhibit the characteristic of having column weights close to zero through task-driven structured sparse training, while avoiding interference from the parameters of the Transformer layer and the visual encoder, and ensuring that the dimensional sparsity is determined only by the semantic adaptability of the word embedding to the task.

[0076] Inputs include a word embedding matrix E containing a pre-trained VLM and a downstream task dataset:

[0077]

[0078] Where N is the sample size. For image data, For the corresponding text label.

[0079] And a pre-trained VLM Transformer text encoder and visual encoder (freeze all parameters and only unlock the gradient update permissions of the word embedding layer).

[0080] Pre-validation requires first verifying the basic performance of the pre-trained word embedding matrix E on downstream tasks, such as fault detection accuracy. ,Require It is no less than 90% of the accuracy rate of human annotation.

[0081] The structured sparse loss function design employs Group Lasso regularization. The loss function is a weighted average of the task loss and the sparse regularization loss, defined as follows:

[0082]

[0083] in, It is the core loss for downstream visual language tasks, and it is adaptively selected according to the task type. If it is a classification task, cross-entropy loss is used, and if it is a cross-modal matching task, contrastive loss is used. It is a loss of mission objectives; It is a sparsity intensity hyperparameter, with a value range of [value range missing]. The optimal value needs to be determined through grid search and testing. Choose to make the proportion of redundant dimensions And task performance loss The minimum λ; E is the word embedding matrix; is the downstream task dataset; i is the dimension index, and the summation term calculates the L2 norm for each column of the word embedding matrix and sums them, forcing the entire column weights of redundant dimensions to shrink synchronously to zero.

[0084] The fine-tuning optimization strategy employs a hierarchical freeze-local update approach, freezing the Transformer text encoder and visual encoder while only unlocking gradient updates for the word embedding layer parameters; it uses the AdamW optimizer with an initial learning rate of... Cosine annealing learning rate scheduling is used; the training period is 1-3 epochs (denoted as...). When the performance of the validation set task shows no improvement for two consecutive rounds or the proportion of redundant dimensions is high... Training stops when the target compression ratio is reached to 1.2 times.

[0085] After fine-tuning, the following must be met:

[0086]

[0087] in, To fine-tune the word embedding matrix, The pre-trained matrix ensures that the core semantic information of the word embedding layer is not excessively corrupted.

[0088] In sparse-induced fine-tuning, stopping conditions are set, including no continuous improvement in validation set task performance and the proportion of redundant dimensions reaching a predetermined multiple of the target compression ratio. Based on these stopping conditions, the training process of sparse-induced fine-tuning is dynamically adjusted and coordinated with the target compression ratio of the generated binary pruning mask to ensure that the sparse-induced results match the subsequent pruning granularity.

[0089] The theoretical advantage of the sparsity-induced fine-tuning module lies in the column-level sparsity of Group Lasso regularization, which ensures that the sparsity-induced results are completely matched with the granularity of the subsequent Transformer layer dimensional pruning, both being dimensional, thus avoiding the fragmentation of intra-dimensional parameters caused by traditional element-level pruning.

[0090] S103. Based on the fine-tuned word embedding matrix, perform dual-indicator quantization to generate a dimension importance score. The dual-indicator quantization includes calculating the weight norm and the ablation statistical significance index.

[0091] Dual-indicator quantification addresses the limitations of single-indicator evaluation by using both parameter-level and task-level indicators for evaluation.

[0092] The first metric is the weight norm (parameter-level importance), which affects the sparsity-induced fine-tuning of the word embedding matrix. Calculate the column L2 norm for each dimension i, as a quantitative indicator of the activity of the parameters in that dimension:

[0093]

[0094] in, The weight norm of dimension i is defined, and it belongs to the non-negative real space. The larger the value, the stronger the contribution of that dimension to the embedding representation of all words.

[0095] right Perform K-means clustering (k=2, cluster centers are...) ,and ), dividing the dimension into candidate important clusters ( , correspond ) and candidate redundant clusters ( , correspond ).

[0096] Calculate the silhouette coefficient, a measure of cluster effectiveness:

[0097]

[0098] in, Let i be the average distance between dimension i and other dimensions within the same cluster. Given the average distance between dimension i and the dimensions of the heterogeneous clusters, we need to find the minimum distance between them. This ensures that the clustering results are significant.

[0099] The second metric is the statistical significance of ablation (task-level importance), which quantifies the contribution of dimension i to task performance through a controlled experiment of masking dimensions and evaluating performance.

[0100] First, in the downstream task validation set The above tests the model performance without masking any dimensions. (e.g., fault detection accuracy); for each dimension i, perform element-level zeroing masking (i.e., ... ), test the performance after masking The performance loss is quantified as follows:

[0101]

[0102]

[0103] in, This represents the task contribution of dimension i. The larger the value, the stronger the necessity of dimension i for the task.

[0104] Repeat the masking experiment n=5 times for each dimension i, and calculate... Mean and 95% confidence interval:

[0105]

[0106]

[0107] in, For n experiments The standard deviation.

[0108] like , ( If the threshold for performance loss is set, then the task importance of dimension i is considered statistically significant; otherwise, it is considered a statistically redundant dimension.

[0109] Cluster analysis was performed on the weight norms to generate dimensional clustering results, which categorized dimensions into important and redundant clusters. Multiple repeated experiments were conducted on the ablation statistical significance index to generate confidence intervals for performance loss. Based on the dimensional clustering results and performance loss confidence intervals, the statistical significance of the dimensional importance scores was verified, and the scores were adjusted to ensure that the evaluation results were comprehensive and free of misjudgments.

[0110] The technical advantage of dual-indicator importance quantification lies in upgrading dimensional importance assessment from qualitative judgment to quantitative statistical verification through multiple repeated experiments and confidence interval analysis, avoiding the randomness error of a single experiment. In the weight norm calculation of indicator one, the L2 norm can effectively reflect the overall amplitude of the entire set of parameters, avoiding misjudgment of dimensional importance by a single extreme parameter. In the ablation statistical significance assessment of indicator two, the performance loss threshold is used to determine statistical significance, ensuring the accurate quantification of dimensional necessity.

[0111] S104. Generate a binary pruning mask based on the dimensional importance score.

[0112] Binary pruning mask generation is achieved through four steps: index normalization, weighted fusion, mask generation, and validity verification.

[0113] First, the indicators are normalized. Since... and The dimensions differ greatly, so Min-Max normalization is used to map them to the [0,1] interval:

[0114]

[0115]

[0116] Among them, if = Then set =0.5, handle similarly. In extreme cases, avoid division by zero errors.

[0117] Then, a weighted fusion of the two indicators is performed, allocating weight coefficients based on the principle of prioritizing parameter-level indicators and supplementing them with task-level indicators. and ( ), calculate the final importance score for each dimension i:

[0118]

[0119] Among them, the weighting coefficients were obtained through 5-fold cross-validation. Optimize and test ( Select to press The combination that yields the best model performance after pruning.

[0120] Next, a binary pruning mask is generated. Based on the target compression ratio r, a binary mask of length H is generated. For all dimensions Sort in descending order to obtain the sort index. Retain the top K most important dimensions (with a mask value of 1), and set the remaining HK dimensions to 0:

[0121]

[0122] Finally, the mask validity is verified by testing the pruned matrix on the validation set. = Corresponding model performance ,Require (in (This is the maximum acceptable performance loss threshold). If this threshold is not met, the compression ratio r is reduced or readjusted. Repeat the steps described above.

[0123] Binary pruning mask generation uses metric normalization to handle extreme cases and ensure numerical stability. The selection of weight coefficients in the weighted fusion stage is based on cross-validation, optimizing the combination to balance the importance of parameter-level and task-level metrics. After mask generation, validity verification uses performance thresholds to ensure pruning safety and avoid sudden performance drops.

[0124] S105. Based on the mapping relationship between the binary pruning mask and the column dimension of the word embedding matrix and the Transformer layer dimension, the Transformer layer is structurally pruned to generate a pruned visual language model.

[0125] Transformer layer pruning is based on a one-to-one mapping between word embedding dimensions (column dimensions of the word embedding matrix) and Transformer layer dimensions.

[0126] The word embedding vectors, as input to the Transformer text encoder, have a dimension H that determines the feature dimension of all subsequent text-side components. The word embedding dimension i The mapping relationship with key parameters of the Transformer layer includes: Q / K / V projection matrix The i-th row corresponds to the input weights of word embedding dimension i, and the i-th column corresponds to the output weights; the self-attention score matrix S depends on the dimension H of q; the Feed-Forward input matrix... The i-th row corresponds to the input weights; the Feed-Forward output matrix The i-th column corresponds to the output weights; cross-modal interaction matrix The i-th row corresponds to the interaction weight.

[0127] When pruning, for the mask For redundant dimension i, delete the i-th row or column of the Transformer layer parameters.

[0128] When performing structured pruning of the Transformer layer based on the mapping relationship between the column dimensions of the binary pruning mask and the word embedding matrix and the Transformer layer dimensions, the mapping relationship between the Transformer layer components and the word embedding dimension i needs to be detailed and explicit.

[0129] The mapping relationship between Transformer layer components and word embedding dimension i is shown in Table 1:

[0130] Table 1

[0131]

[0132] The mapping relationship between word embedding dimension i (i ∈ [1, H]) and key parameters of the Transformer layer is as follows: For the Q / K / V projection matrix ( The shape of the unpruned parameters is The mapping relationship between the matrix and the word embedding dimension i is that the i-th row of the matrix corresponds to the input weight of the word embedding dimension i, and the i-th column corresponds to the output weight of the word embedding dimension i. The pruning basis is that when the mask M[i]=0, the i-th row (input side) and the i-th column (output side) are deleted.

[0133] For the self-attention score matrix S, the shape of the unpruned parameters is... (L is the sequence length), and the mapping relationship with the word embedding dimension i is that the score S[m,n] depends on the dimension H of the query vector q. The pruning basis is that there is no direct pruning, but the computation is indirectly reduced due to the reduction of dimension.

[0134] For the Feed-Forward input matrix ( , (For intermediate layer dimensions), the shape of the unpruned parameters is The mapping relationship between M[i] and word embedding dimension i is the input weight of word embedding dimension i corresponding to the i-th row of the matrix. The pruning criterion is to delete the i-th row when M[i]=0.

[0135] For the Feed-Forward output matrix ( The shape of the unpruned parameters is The mapping relationship between M[i] and word embedding dimension i is the output weight of word embedding dimension i corresponding to the i-th column of the matrix. The pruning criterion is to delete the i-th column when M[i]=0.

[0136] For cross-modal interaction matrix ( (C is the interaction feature dimension), the shape of the unpruned parameter is The mapping relationship between M[i] and word embedding dimension i is the interaction weight of word embedding dimension i corresponding to the i-th row of the matrix. The pruning criterion is to delete the i-th row when M[i]=0.

[0137] This mapping ensures dimensional consistency during pruning, avoiding architectural errors.

[0138] A two-dimensional diagonal mask matrix corresponding to the binary pruning mask is constructed. The two-dimensional diagonal mask matrix is ​​then used to perform matrix multiplication on the query, key, and value projection matrices of the Transformer layer to achieve structured pruning. Specifically, a two-dimensional diagonal mask matrix is ​​constructed based on the one-dimensional mask M. Pruning transformation is performed on the Q / K / V projection matrix:

[0139]

[0140]

[0141]

[0142] The dimension of the pruned matrix is... (K is the number of dimensions to be retained).

[0143] The computational complexity of self-attention is unpruned. Reduced to the level after pruning (L is the sequence length), the complexity reduction ratio is:

[0144]

[0145] For multi-head attention, the single-head pruning logic needs to be extended to multi-head scenarios:

[0146] First, project the unpruned Q matrix. Split into h single-head projection matrices (h is the number of heads, Then, for each single-head projection matrix, delete the i-th row (M[i]=0) according to the dimensional mapping relationship to obtain... ( Then, merge the h pruned single-head matrices to obtain the Q-projection matrix after multi-head pruning. This ensures that the parallel computation logic of multi-head attention remains unchanged.

[0147] The classic structure of a feed-forward layer is "linear transformation → activation function → linear transformation", that is:

[0148]

[0149] in These are input features. It is the input matrix. It is the output matrix. For GELU activation function, This is a bias term.

[0150] Input matrix pruning, The Line-corresponding word embedding dimension The input weights are used to delete the i-th row during pruning, resulting in... :

[0151]

[0152] Input features after pruning The result of the linear transformation is , and unpruned branches Apply mask Consistent results ensure the correct input dimension for the activation function.

[0153] Output matrix pruning, The output weights of the i-th column corresponding to the word embedding dimension i are obtained by deleting the i-th column during pruning. :

[0154]

[0155] After pruning, the FFN output is The dimensions are consistent with the dimensions of the pruned input features, thus avoiding dimension mismatch during residual connections.

[0156] Bias term processing, bias term (No dimensional correlation, no pruning required) (The i-th element needs to be deleted) =0), get : .

[0157] In the dimensional pruning of the Transformer linear layer, the parameter compression effect of the Feed-Forward layer pruning can be quantified:

[0158] The parameters for the unpruned branches are:

[0159]

[0160]

[0161] The parameters after pruning are:

[0162]

[0163]

[0164] when (like When the compression ratio is ), Approximately:

[0165]

[0166] In cross-modal interaction layer pruning, after the interaction matrix is ​​pruned, the dimensions of text features and visual features are consistent, and element-level interactions can be performed normally.

[0167] S106. Through knowledge distillation, the performance of the pruned visual language model is restored using the dimensional projection matrix to obtain the final pruned visual language model.

[0168] Knowledge distillation is achieved through a teacher-student framework, where the teacher model T uses an unpruned, pre-trained VLM, and the student model S uses a pruned, compact VLM.

[0169] Teacher model T is defined as:

[0170]

[0171] in, For image features, t represents the output dimension of the visual encoder; t is the text label, which becomes after word embedding. ; Output the visual features of the teacher model; This outputs the text features of the teacher model.

[0172] The student model S is defined as:

[0173]

[0174] Among them, text word embedding is (using binary pruning mask) filter); ; Output the text features for the student model.

[0175] To ensure knowledge transferability, hierarchical alignment is required, meaning the teacher and student models must have the same number of Transformer layers, and the number of attention heads in the l-th layer (l=1,2,…,L) must be the same and h, guaranteeing the comparability of knowledge at the same layer. Simultaneously, dimensional mapping must be satisfied, introducing a learnable dimensional projection matrix. This involves mapping the H-dimensional features of the teacher model to the K-dimensional space of the student model, thus resolving the dimensionality differences caused by pruning.

[0176]

[0177] in, The mapped teacher text features, and student features Dimensions are consistent.

[0178] Hierarchical knowledge distillation includes attention-layer knowledge distillation, linear-layer knowledge distillation, and cross-modal alignment distillation. In attention-layer knowledge distillation, the core knowledge of the attention layer is the semantic association weights, which need to be transferred through the distribution consistency constraint of Q / K / V features. Since the teacher's Q / K / V features are H-dimensional, single-head... The student is K-dimensional, single-headed. First, dimensional alignment needs to be achieved through in-head projection.

[0179] Q-features of the teacher at level l, head j (L is the sequence length), through the head projection matrix Mapping to the student single-head dimension, the K / V features are similarly introduced, respectively. :

[0180]

[0181] in, The Q-features of the teacher after mapping.

[0182] Attention feature distillation loss calculates the mean squared error (MSE) of Q / K / V features for students and mapped teachers, constraining the consistency of feature distributions within the same layer and among the same teachers.

[0183]

[0184] in, For the Q-feature of the student at the l-th level and the j-th head, the , where n is the total number of characteristic elements.

[0185] The total attention loss across layers is a weighted sum of the attention losses over all L layers, with slightly higher weights for shallower layers because shallower layers preserve the basic semantics.

[0186]

[0187] In linear layer (FFN) knowledge distillation, the core knowledge of the FFN layer is the ability to transform features nonlinearly. Its output is a real-valued feature vector, and the maximum mean difference (MMD) is used to measure the distribution difference of continuous features.

[0188] The teacher's output of the l-th layer FFN is After dimensional projection, it becomes ;

[0189] The student's output at level l, FFN, is .

[0190] FFN distillation loss is measured using MMD. and Distribution differences:

[0191]

[0192] Where N is the batch size. The FFN output for the i-th student sample is given, and the kernel function is a Gaussian kernel. .

[0193] The total FFN loss across layers is a weighted sum, with slightly higher weights for deeper layers because the features at deeper layers are more abstract.

[0194]

[0195]

[0196]

[0197] In cross-modal alignment distillation, the core knowledge of cross-modal alignment is the semantic matching degree of image-text pairs, which is achieved through the consistency constraint of the similarity matrix.

[0198] The teacher model calculates the cosine similarity between image features and text features, obtaining a matrix. N is the number of samples in a batch:

[0199]

[0200] in, For teachers' visual characteristics, Features of teacher-written texts.

[0201] The student model is obtained similarly. Text features are Their visual characteristics are consistent with those of the teacher.

[0202] When calculating cross-modal distillation loss, to enhance the guidance of soft tags, the following steps are first performed: Temperature coefficient softening ( =0.1, magnify the difference), then calculate with MSE:

[0203]

[0204] Joint fine-tuning optimization uses a multi-objective loss function to jointly optimize the student model, balancing task adaptability and knowledge inheritance.

[0205] The total loss is a weighted average of the downstream task loss and the stratified distillation loss:

[0206]

[0207] in, The weights represent task loss, ensuring that the student model adapts to downstream tasks rather than simply imitating the teacher. , , Represents the weight of distillation loss, satisfying ,generally Cross-modal alignment is the core performance indicator; It is the core loss for downstream visual language tasks, consistent with the loss in sparse-induced fine-tuning.

[0208] Fine-tuning strategies and convergence criteria include:

[0209] The optimizer configuration uses the AdamW optimizer with an initial learning rate of Weight decay is set to The training cycle consists of 2-3 epochs, employing an early stopping strategy. If the performance on the validation set shows no improvement for 3 consecutive epochs, training is stopped.

[0210] The parameter update scope only updates the Transformer layer parameters and dimensional projection matrix of the student model. Freeze the visual encoder and word embedding layer.

[0211] After fine-tuning, the performance recovery effect should be verified through the following indicators:

[0212] Task performance recovery rate:

[0213]

[0214] Require .

[0215] in, For student model performance, For teacher model performance;

[0216] Parameter efficiency ratio:

[0217]

[0218] It is necessary to maintain the same compression rate as the pruning target.

[0219] Cross-modal alignment error:

[0220]

[0221] F represents the Frobenius norm ratio, requiring This ensures that the similarity matrices of students and teachers are highly consistent.

[0222] By using hierarchical knowledge distillation and dimensional adaptive mapping, the problem of dimensionality difference between the pruned model and the original model is solved, and the semantic association features of the intermediate layer can be recovered more accurately.

[0223] Please refer to Figure 5 As shown, this application also provides a perceptual word embedding dimension dynamic pruning device, including:

[0224] Module 201 obtains the word embedding matrix based on the word embedding layer of the pre-trained visual language model;

[0225] Fine-tuning module 202 performs sparse-induced fine-tuning on the word embedding matrix to generate a fine-tuned word embedding matrix. The sparse-induced fine-tuning includes task-driven structured sparse training using Group Lasso regularization.

[0226] The scoring module 203 performs dual-indicator quantization based on the fine-tuned word embedding matrix to generate a dimension importance score. The dual-indicator quantization includes calculating the weight norm and the ablation statistical significance index.

[0227] Masking module 204 generates a binary pruned mask based on the dimensional importance score;

[0228] Model module 205 performs structured pruning on the Transformer layer based on the binary pruning mask and the mapping relationship between the column dimensions of the word embedding matrix and the Transformer layer dimensions, generating a pruned visual language model.

[0229] The recovery module 206 performs performance recovery on the pruned visual language model by using a dimensional projection matrix through knowledge distillation to obtain the final pruned visual language model.

[0230] Optionally, the scoring module 203 performs dual-index quantization based on the fine-tuned word embedding matrix to generate a dimensional importance score, including:

[0231] Cluster analysis is performed on the weight norm to generate dimension clustering results, wherein the cluster analysis divides the dimensions into important clusters and redundant clusters;

[0232] The statistical significance index of ablation was tested repeatedly to generate confidence intervals for performance loss.

[0233] Based on the dimensional clustering results and the performance loss confidence interval, the statistical significance of the dimensional importance score is verified, and the dimensional importance score is adjusted based on the statistical significance.

[0234] Optionally, the fine-tuning module 202 performs sparse-induced fine-tuning on the word embedding matrix to generate a fine-tuned word embedding matrix, including:

[0235] In the sparse-induced fine-tuning, a stopping condition is set, which includes no continuous improvement in the performance of the validation set task and the proportion of redundant dimensions reaching a predetermined multiple of the target compression rate.

[0236] The training process of the sparse-induced fine-tuning is dynamically adjusted according to the stopping condition, in coordination with the target compression rate of the generated binary pruning mask.

[0237] Optionally, the model module 205, based on the mapping relationship between the binary pruning mask and the column dimensions of the word embedding matrix and the Transformer layer dimensions, performs structured pruning on the Transformer layer to generate a pruned visual language model, including:

[0238] Construct a two-dimensional diagonal mask matrix corresponding to the binary pruning mask;

[0239] The two-dimensional diagonal mask matrix is ​​used to perform matrix multiplication on the query, key, and value projection matrices of the Transformer layer to achieve structured pruning.

[0240] Optionally, the recovery module 206 performs performance recovery on the pruned visual language model using a dimensional projection matrix through knowledge distillation, including:

[0241] In attention layer distillation, a learnable projection matrix is ​​introduced for each head;

[0242] The learnable projection matrix is ​​used to perform dimensional mapping on the query, key, and value features of the teacher model to generate aligned attention features;

[0243] Based on the aligned attention features, cross-modal alignment distillation is performed, including: calculating a loss function through a temperature-softened similarity matrix.

[0244] Optionally, the scoring module 203 performs dual-index quantization based on the fine-tuned word embedding matrix to generate a dimensional importance score, including:

[0245] The loss weights in the knowledge distillation are adjusted based on the importance scores of the aforementioned dimensions.

[0246] The loss weights are adaptively optimized using a validation set, forming a feedback loop between the dual-index quantization and the knowledge distillation.

[0247] Optionally, the fine-tuning module 202 performs sparse-induced fine-tuning on the word embedding matrix to generate a fine-tuned word embedding matrix, including:

[0248] The sparse-induced fine-tuning and the knowledge distillation are jointly optimized using a multi-objective loss function;

[0249] In the joint optimization, the stopping condition of the sparse-induced fine-tuning optimizes the dimensional projection matrix learning of the knowledge distillation.

[0250] This application also provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0251] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the above-described method.

[0252] The above description of the embodiments is provided to enable those skilled in the art to understand and apply this application. Those skilled in the art will readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without inventive effort. Therefore, this application is not limited to the above embodiments, and any improvements and modifications made to this application based on the disclosure thereof should be within the scope of protection of this application.

Claims

1. A perceptual word embedding dimension dynamic pruning method, characterized in that, include: Obtain the word embedding matrix based on the word embedding layer of the pre-trained visual language model; The word embedding matrix is ​​subjected to sparse-induced fine-tuning to generate a fine-tuned word embedding matrix. The sparse-induced fine-tuning includes task-driven structured sparse training using Group Lasso regularization. Based on the fine-tuned word embedding matrix, dual-indicator quantization is performed to generate dimensional importance scores. The dual-indicator quantization includes calculating the weight norm and the ablation statistical significance index. Generate a binary pruning mask based on the aforementioned dimensional importance score; Based on the binary pruning mask and the mapping relationship between the column dimension of the fine-tuned word embedding matrix and the dimension of the Transformer layer, the Transformer layer is structurally pruned to generate a pruned visual language model. By using knowledge distillation and employing a dimensional projection matrix, the performance of the pruned visual language model is restored, resulting in the final pruned visual language model.

2. The method according to claim 1, characterized in that, Based on the fine-tuned word embedding matrix, dual-index quantization is performed to generate dimensional importance scores, including: Cluster analysis is performed on the weight norm to generate dimension clustering results, wherein the cluster analysis divides the dimensions into important clusters and redundant clusters; The statistical significance index of ablation was tested repeatedly to generate confidence intervals for performance loss. Based on the dimensional clustering results and the performance loss confidence interval, the statistical significance of the dimensional importance score is verified, and the dimensional importance score is adjusted based on the statistical significance.

3. The method according to claim 1, characterized in that, The word embedding matrix is ​​subjected to sparse-induced fine-tuning to generate a fine-tuned word embedding matrix, including: In the sparse-induced fine-tuning, a stopping condition is set, which includes no continuous improvement in the performance of the validation set task and the proportion of redundant dimensions reaching a predetermined multiple of the target compression rate. The training process of the sparse-induced fine-tuning is dynamically adjusted according to the stopping condition, in coordination with the target compression rate of the generated binary pruning mask.

4. The method according to claim 1, characterized in that, Based on the mapping relationship between the binary pruning mask and the column dimensions of the word embedding matrix and the Transformer layer dimensions, the Transformer layer is structurally pruned to generate a pruned visual language model, including: Construct a two-dimensional diagonal mask matrix corresponding to the binary pruning mask; The two-dimensional diagonal mask matrix is ​​used to perform matrix multiplication on the query, key, and value projection matrices of the Transformer layer to achieve structured pruning.

5. The method according to claim 1, characterized in that, Through knowledge distillation, the performance of the pruned visual language model is restored using a dimensional projection matrix, including: In attention layer distillation, a learnable projection matrix is ​​introduced for each head; The learnable projection matrix is ​​used to perform dimensional mapping on the query, key, and value features of the teacher model to generate aligned attention features; Based on the aligned attention features, cross-modal alignment distillation is performed, including: calculating a loss function through a temperature-softened similarity matrix.

6. The method according to claim 1, characterized in that, Based on the fine-tuned word embedding matrix, dual-index quantization is performed to generate dimensional importance scores, including: The loss weights in the knowledge distillation are adjusted based on the importance scores of the aforementioned dimensions. The loss weights are adaptively optimized using a validation set, forming a feedback loop between the dual-index quantization and the knowledge distillation.

7. The method according to claim 1, characterized in that, The word embedding matrix is ​​subjected to sparse-induced fine-tuning to generate a fine-tuned word embedding matrix, including: The sparse-induced fine-tuning and the knowledge distillation are jointly optimized using a multi-objective loss function; In the joint optimization, the stopping condition of the sparse-induced fine-tuning optimizes the dimensional projection matrix learning of the knowledge distillation.

8. A perceptual word embedding dimension dynamic pruning device, characterized in that, include: The acquisition module obtains the word embedding matrix based on the word embedding layer of the pre-trained visual language model; The fine-tuning module performs sparse-induced fine-tuning on the word embedding matrix to generate a fine-tuned word embedding matrix. The sparse-induced fine-tuning includes task-driven structured sparse training using Group Lasso regularization. The scoring module performs dual-indicator quantization based on the fine-tuned word embedding matrix to generate a dimension importance score. The dual-indicator quantization includes calculating the weight norm and the ablation statistical significance index. The masking module generates a binary pruned mask based on the dimensional importance score. The model module performs structured pruning on the Transformer layer based on the binary pruning mask and the mapping relationship between the column dimensions of the word embedding matrix and the Transformer layer dimensions, generating a pruned visual language model. The recovery module performs performance recovery on the pruned visual language model by using knowledge distillation and a dimensional projection matrix to obtain the final pruned visual language model.

9. The apparatus according to claim 8, characterized in that, The scoring module performs dual-index quantization based on the fine-tuned word embedding matrix to generate dimensional importance scores, including: Cluster analysis is performed on the weight norm to generate dimension clustering results, wherein the cluster analysis divides the dimensions into important clusters and redundant clusters; The statistical significance index of ablation was tested repeatedly to generate confidence intervals for performance loss. Based on the dimensional clustering results and the performance loss confidence interval, the statistical significance of the dimensional importance score is verified, and the dimensional importance score is adjusted based on the statistical significance.

10. The apparatus according to claim 8, characterized in that, The fine-tuning module performs sparse-induced fine-tuning on the word embedding matrix to generate a fine-tuned word embedding matrix, including: In the sparse-induced fine-tuning, a stopping condition is set, which includes no continuous improvement in the performance of the validation set task and the proportion of redundant dimensions reaching a predetermined multiple of the target compression rate. The training process of the sparse-induced fine-tuning is dynamically adjusted according to the stopping condition, in coordination with the target compression rate of the generated binary pruning mask.

11. The apparatus according to claim 8, characterized in that, The model module, based on the mapping relationship between the binary pruning mask and the column dimensions of the word embedding matrix and the Transformer layer dimensions, performs structured pruning on the Transformer layer to generate a pruned visual language model, including: Construct a two-dimensional diagonal mask matrix corresponding to the binary pruning mask; The two-dimensional diagonal mask matrix is ​​used to perform matrix multiplication on the query, key, and value projection matrices of the Transformer layer to achieve structured pruning.

12. The apparatus according to claim 8, characterized in that, The recovery module performs performance recovery on the pruned visual language model using knowledge distillation and a dimensional projection matrix, including: In attention layer distillation, a learnable projection matrix is ​​introduced for each head; The learnable projection matrix is ​​used to perform dimensional mapping on the query, key, and value features of the teacher model to generate aligned attention features; Based on the aligned attention features, cross-modal alignment distillation is performed, including: calculating a loss function through a temperature-softened similarity matrix.

13. The apparatus according to claim 8, characterized in that, The scoring module performs dual-index quantization based on the fine-tuned word embedding matrix to generate dimensional importance scores, including: The loss weights in the knowledge distillation are adjusted based on the importance scores of the aforementioned dimensions. The loss weights are adaptively optimized using a validation set, forming a feedback loop between the dual-index quantization and the knowledge distillation.

14. The apparatus according to claim 8, characterized in that, The fine-tuning module performs sparse-induced fine-tuning on the word embedding matrix to generate a fine-tuned word embedding matrix, including: The sparse-induced fine-tuning and the knowledge distillation are jointly optimized using a multi-objective loss function; In the joint optimization, the stopping condition of the sparse-induced fine-tuning optimizes the dimensional projection matrix learning of the knowledge distillation.

15. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed in a computer, causes the computer to perform the method described in any one of claims 1-7.

Images