Task-adaptive visual language large model collaborative pruning method

Abstract

The present application relates to the field of language model pruning, and specifically discloses a task-adaptive visual language large model collaborative pruning method, comprising: S1, task perception, obtaining a training data set containing image text pairs, shielding the corresponding data set of a single mode, calculating the loss value and the mode dependency, and adaptively biasing the visual and language module share; S2, calculating the importance score, dividing the model weight, calculating the gradient norm approximation in layers, multiplying the mode share to obtain the score of each layer; S3, allocating sparsity, normalizing the score, calculating the sparsity of each layer combined with the total parameter retention ratio and setting a threshold; S4, structured sparsity, combining pruning strategy and token screening, calculating the fusion score of mode correlation, calculating the channel score by retaining effective tokens, and retaining key channels according to groups. The technical scheme provided by the present application solves the problems of poor adaptation of traditional pruning, not considering task mode dependency, and easy mis-pruning of key parameters, improves compression and inference efficiency, avoids cross-modal information loss, and reduces performance loss.

Description

Technical Field

[0001] This invention relates to the field of language model pruning technology, specifically a task-adaptive collaborative pruning method for large visual language models. Background Technology

[0002] Collaborative pruning techniques for large-scale visual-language models aim to optimize the redundant parameter pruning process by modeling the inherent dependencies between visual and language modalities. This effectively improves inference speed and deployment efficiency while maintaining cross-modal semantic alignment. However, this technique faces three core challenges: First, computational resource overhead. While structured pruning maintains a hardware-friendly rule structure, globally searching for the optimal pruning structure consumes significant computational power. Second, modal adaptability. The features of visual and language models in large-scale visual-language models are heterogeneous, making it difficult for traditional pruning methods to balance the redundancy removal of key components with modal alignment requirements. Third, insufficient utilization of multimodal information. The modal interaction complexity of different input examples in visual-language models varies significantly, and traditional pruning methods do not consider the degree of modal requirements of different inputs.

[0003] Existing pruning methods are categorized into two technical paradigms: structured pruning and unstructured pruning. Structured pruning performs progressive layer or channel pruning by uniformly searching for the optimal pruning ratio among the visual encoder, text decoder, and cross-modal connector, and recovers performance through fine-tuning. Unstructured pruning focuses on weight sparsity, achieving fine-grained pruning by globally allocating sparsity across layers. However, these traditional methods have significant drawbacks:

[0004] Most pruning methods still use single-modal importance assessment criteria (such as gradient, absolute weight, etc.) to prune models. This method does not adapt to the unique modal attributes of large visual-language models. Pruning based solely on visual or linguistic saliency features can lead to cross-modal feature degradation.

[0005] Traditional methods do not adequately analyze the modal dependency differences among various downstream tasks. For example, in vision-driven downstream task pruning, excessive retention of language model parameters can lead to a waste of computational resources.

[0006] Traditional Wanda pruning does not incorporate token filtering or cross-modal interaction information, resulting in noise in the activation statistics and leading to the problem of incorrect pruning of key cross-modal parameters.

[0007] Therefore, how to propose a collaborative pruning method based on downstream task awareness for the field of large visual language model compression, which can significantly reduce model size while improving inference efficiency and accurately match the modal requirements of downstream tasks to avoid performance loss, has become an important problem that needs to be solved in the field of large visual language model pruning. Summary of the Invention

[0008] To address the aforementioned problems in existing technologies, this invention provides a task-adaptive collaborative pruning method for large visual language models. This method solves the problems of cross-modal feature degradation, failure to adapt to downstream task modal dependencies, and easy mispruning of cross-modal key parameters in traditional pruning, effectively improving model compression efficiency and inference speed, and ensuring the performance of the pruned model.

[0009] To achieve the above objectives, this invention proposes a task-adaptive collaborative pruning method for large visual language models, comprising:

[0010] S1. Task Awareness: Obtain the training dataset for the multimodal downstream task, which contains multiple image-text pairs. By masking the single modal information in the training dataset, obtain image-text pairs that mask visual information and image-text pairs that mask language information. Calculate the loss value when masking a single modal input using the above image-text pairs. Determine the modal dependency of the downstream task based on the loss value. Introduce adaptive bias fine-tuning of the share allocation of the visual module and the language module to obtain the visual share and text share corresponding to the downstream task.

[0011] S2. Importance score calculation: The weights of the visual-language large model are divided into visual model weights and language model weights. The gradient norm approximation values ​​of the visual model layer and the language model layer are calculated using hierarchical weights. The gradient norm of each layer is multiplied by the share of the corresponding modality to obtain the importance scores of each layer of the visual module and each layer of the language module.

[0012] S3. Sparsity Allocation: Summarize the importance scores of the visual model and the language model, calculate the normalized proportion of the importance scores of each layer using a normalization method; based on the normalization proportion and the preset total parameter retention ratio, calculate the sparsity of each layer of the visual model and each layer of the language model, and set a maximum sparsity threshold for each layer.

[0013] S4. Structured Sparsity for Token Filtering: Based on a pruning strategy using the product of absolute weight and activation value, combined with a token filtering mechanism guided by cross-modal attention weights, redundant parameters within each layer are pruned. First, the modal correlation of each token is calculated using bidirectional cross-modal attention weights, and the cross-modal fusion filtering score is obtained by combining it with the initial single-modal score. Tokens with valid fusion scores are retained. The norm of the activation channel is calculated based on the valid tokens, and the importance score of the input channel is obtained by combining it with the absolute weight. The input channels are sorted according to their importance scores, and a 2:4 structured sparsity pattern is formed by deleting two channels in every group of four, retaining the core parameter presets and pruning redundant channels.

[0014] Preferably, in S1, the specific formula for calculating the loss value when shielding a single modal input is as follows:

[0015] ;

[0016] ;

[0017] In the formula, Representing visual modality, Representing language modality, The original training dataset, To mask the dataset after the corresponding modality, This represents the mean loss value selected from the training dataset for each training sample. For the entire visual language model, This is the model loss function.

[0018] Preferably, in S1, the formula for the initial allocation of visual module shares is:

[0019] ;

[0020] In the formula, For the initial visual module share, To mask the loss value when the visual modality is not displayed, The loss value when language modalities are masked;

[0021] Introducing adaptive bias The formulas for calculating the visual share and text share are as follows:

[0022] ;

[0023] ;

[0024] In the formula, For the final visual share, This is the final text share.

[0025] Preferred, adaptive bias The calculation formula is:

[0026] ;

[0027] In the formula, Parameters for controlling the adaptive bias size.

[0028] Preferably, in S2, the formula for calculating the approximate value of the gradient norm is:

[0029] ;

[0030] In the formula, For the first The target weights of the layer, where D is the cross-modal calibration sample set. To calibrate visual-text pairs in a cross-modal calibration sample set, The number of noise samples per sample. For the disturbance amplitude to exceed the parameter, This represents the loss difference due to bidirectional disturbances.

[0031] Preferably, in S2, the formulas for calculating the importance scores of the visual model layer and the language model layer are as follows:

[0032] ;

[0033] In the formula, For the visual module Layer importance score, For the language module, layer 1 Importance score, For the visual model Layer weights, For the language model Layer weights, It is an L2 norm.

[0034] Preferably, in S3, the formula for calculating the sparsity of each layer is:

[0035] ;

[0036] In the formula, For the visual model The sparsity of the layer, For the language model The sparsity of the layer, For the corresponding mode number The normalized percentage of layer importance scores, , The preset percentage for retaining total parameters, where |W| represents the total number of parameters in the model. For the visual model The number of parameters in the layer For the language model The number of parameters in a layer.

[0037] Preferably, in S4, the formula for calculating the importance score of the input channel is:

[0038] ;

[0039] In the formula, It is the in-layer weight matrix. It is the number of output channels. This is the number of input channels. Indicates the first The absolute values ​​of the elements of the input channel weight vectors are taken and then averaged. For the first Importance scores for each input channel For the first The average of the absolute values ​​of the elements of the input channel weight vector. For the corresponding number in the activation matrix The activation vectors for each input channel, and the activation matrix has dimensions of . , , It refers to the batch size. Given the length of the input sequence, Total number of tokens For the corresponding number in the activation matrix The activation vectors of the input channels, Let L2 be the norm of the activation vector.

[0040] Preferably, in S4, the formula for calculating the bidirectional cross-modal attention weights is:

[0041] ;

[0042] ;

[0043] In the formula, The attention weight of the visual token over the text token. The attention weight of the text token on the visual token. A query matrix for visual tokens. The query matrix for text tokens. A key matrix for visual tokens. The key matrix of text tokens. The attention head dimension is denoted by 'attention head', and 'softmax' is the normalization function.

[0044] Preferably, in S4, the formula for calculating the cross-modal fusion screening score is:

[0045] ;

[0046] ;

[0047] In the formula, For cross-modal fusion of visual tokens, a score is selected. Filter scores for cross-modal fusion of text tokens. The initial score for the single-modal visual token. The initial score for the unimodal nature of the text token. The cross-modal attention weights are after batch averaging. The cross-modal attention weights are the result of averaging across multiple dimensions.

[0048] The visual token's initial unimodal score Unimodal initial score of text token The calculation formula is:

[0049] ;

[0050] .

[0051] Therefore, this invention proposes a task-adaptive collaborative pruning method for large visual language models, with the following beneficial effects:

[0052] (1) The importance score is calculated by combining the gradient and the modal loss of the downstream task. Only forward propagation is needed to complete the calculation, which reduces the memory threshold and avoids cross-modal feature degradation caused by single-modal pruning, thus ensuring model performance.

[0053] (2) By analyzing modal dependencies through downstream task perception, pruning sparsity is allocated according to task requirements to avoid excessive retention of single modal parameters, reduce computing power and memory waste, and achieve lightweight and efficient model.

[0054] (3) Adopting a structured sparse pattern to adapt to mainstream hardware, incorporating cross-modal token screening, purification and activation statistics, accurately retaining cross-modal key parameters, and balancing the model's lightweightness and cross-modal task adaptability.

[0055] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0056] Figure 1 This is the algorithm flowchart of the task-adaptive collaborative pruning method for large visual language models of the present invention;

[0057] Figure 2 This is the sparsity transformation diagram of the task-adaptive collaborative pruning method for large visual language models of this invention;

[0058] Figure 3 This is a flowchart of the cross-modal attention weight-guided token selection mechanism of the task-adaptive visual language large model collaborative pruning method of this invention;

[0059] Figure 4 This is the weight absolute value-activation value product pruning graph of the "2:4" structured sparse pattern of the collaborative pruning method for large visual language models in this invention. Detailed Implementation

[0060] To make the technical solutions, advantages, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below. The described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the protection scope of this application.

[0061] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0062] like Figures 1-4 As shown, the task-adaptive collaborative pruning method for large visual language models provided by this invention includes:

[0063] S1. Task Awareness: Obtain the training dataset for the multimodal downstream task, which contains multiple image-text pairs. By masking the single modal information in the training dataset, obtain image-text pairs that mask visual information and image-text pairs that mask language information. Calculate the loss value when masking a single modal input using the above image-text pairs. Determine the modal dependency of the downstream task based on the loss value. Introduce adaptive bias fine-tuning of the share allocation of the visual module and the language module to obtain the visual share and text share corresponding to the downstream task.

[0064] The specific formula for calculating the loss value when masking a single modal input is as follows:

[0065] ;

[0066] ;

[0067] In the formula, Representing visual modality, Representing language modality, The original training dataset, To mask the dataset after the corresponding modality, This represents the mean loss value selected from the training dataset for each training sample. For the entire visual language model, This is the model loss function.

[0068] The formula for the initial allocation of visual module shares is:

[0069] ;

[0070] In the formula, For the initial visual module share, To mask the loss value when the visual modality is not displayed, The loss value when language modalities are masked;

[0071] Adaptive bias The calculation formula is:

[0072] ;

[0073] In the formula, Parameters for controlling the size of the adaptive bias;

[0074] Introducing adaptive bias The formulas for calculating the visual share and text share are as follows:

[0075] ;

[0076] ;

[0077] In the formula, For the final visual share, This is the final text share.

[0078] S2. Importance score calculation: The weights of the visual-language large model are divided into visual model weights and language model weights. The gradient norm approximation values ​​of the visual model layer and the language model layer are calculated using hierarchical weights. The gradient norm of each layer is multiplied by the share of the corresponding modality to obtain the importance scores of each layer of the visual module and each layer of the language module.

[0079] The formula for calculating the approximate value of the gradient norm is:

[0080] ;

[0081] In the formula, For the first The target weights of the layer, where D is the cross-modal calibration sample set. To calibrate visual-text pairs in a cross-modal calibration sample set, The number of noise samples per sample. For the disturbance amplitude to exceed the parameter, This represents the loss difference due to bidirectional disturbances.

[0082] The formulas for calculating the importance scores of the visual model layer and the language model layer are as follows:

[0083] ;

[0084] In the formula, For the visual module Layer importance score, For the language module, layer 1 Importance score, For the visual model Layer weights, For the language model Layer weights, It is an L2 norm.

[0085] S3. Sparsity Allocation: Summarize the importance scores of the visual model and the language model, calculate the normalized proportion of the importance scores of each layer using a normalization method; based on the normalization proportion and the preset total parameter retention ratio, calculate the sparsity of each layer of the visual model and each layer of the language model, and set a maximum sparsity threshold for each layer.

[0086] The formulas for calculating the sparsity of each layer are as follows:

[0087] ;

[0088] In the formula, For the visual model The sparsity of the layer, For the language model The sparsity of the layer, For the corresponding mode number The normalized percentage of layer importance scores, , The preset percentage for retaining total parameters, where |W| represents the total number of parameters in the model. For the visual model The number of parameters in the layer For the language model The number of parameters in a layer.

[0089] S4. Structured Sparsity for Token Filtering: Based on a pruning strategy using the product of absolute weight and activation value, combined with a token filtering mechanism guided by cross-modal attention weights, redundant parameters within each layer are pruned. First, the modal correlation of each token is calculated using bidirectional cross-modal attention weights, and the cross-modal fusion filtering score is obtained by combining it with the initial single-modal score. Tokens with valid fusion scores are retained. The norm of the activation channel is calculated based on the valid tokens, and the importance score of the input channel is obtained by combining it with the absolute weight. The input channels are sorted according to their importance scores, and a 2:4 structured sparsity pattern is formed by deleting two channels in every group of four, retaining the core parameter presets and pruning redundant channels.

[0090] The formula for calculating the importance score of the input channel is:

[0091] ;

[0092] In the formula, It is the in-layer weight matrix. It is the number of output channels. This is the number of input channels. Indicates the first The absolute values ​​of the elements of the input channel weight vectors are taken and then averaged. For the first Importance scores for each input channel For the first The average of the absolute values ​​of the elements of the input channel weight vector. For the corresponding number in the activation matrix The activation vectors for each input channel, and the activation matrix has dimensions of . , , It refers to the batch size. Given the length of the input sequence, Total number of tokens For the corresponding number in the activation matrix The activation vectors of the input channels, Let L2 be the norm of the activation vector.

[0093] The formula for calculating bidirectional cross-modal attention weights is:

[0094] ;

[0095] ;

[0096] In the formula, The attention weight of the visual token over the text token. The attention weight of the text token on the visual token. A query matrix for visual tokens. The query matrix for text tokens. A key matrix for visual tokens. The key matrix of text tokens. The attention head dimension is denoted by 'attention head', and 'softmax' is the normalization function.

[0097] The formula for calculating the cross-modal fusion screening score is:

[0098] ;

[0099] ;

[0100] In the formula, For cross-modal fusion of visual tokens, a score is selected. Filter scores for cross-modal fusion of text tokens. The initial score for the single-modal visual token. The initial score for the unimodal nature of the text token. The cross-modal attention weights are after batch averaging. The cross-modal attention weights are the result of averaging across multiple dimensions.

[0101] Unimodal initial score of visual token Unimodal initial score of text token The calculation formula is:

[0102] ;

[0103] .

[0104] Example 1

[0105] like Figures 1-4 As shown, this embodiment is based on the BLIP2 visual language large model (visual module: ViT-L / 14, language module: T5), and verifies the effectiveness of the "task-adaptive visual language large model collaborative pruning method" for five mainstream multimodal downstream tasks: coco, vqav2, nocaps, gqa, and okvqa.

[0106] The core objectives include:

[0107] 1) The memory usage of the model after pruning is ≤0.7GB, which is much lower than that of traditional methods;

[0108] 2) The cut model can be directly embedded into embedded hardware without the need for large computing devices;

[0109] 3) Ensure that the performance loss of the pruned model is ≤4% in all tasks, balancing lightweight design and practicality;

[0110] 4) Verify the independent effect of the modal perception step on the BLIP small model.

[0111] I. Basic Experimental Setup:

[0112] Hardware environment:

[0113] Pruning training equipment: a single server (CPU: Intel Xeon 8375C 24 cores, GPU: NVIDIA A10 24GB VRAM, Memory: 128GB DDR4) to ensure resource supply during model training and pruning.

[0114] Embedded deployment devices: Embedded edge devices (CPU: ARM Cortex-A788 core, embedded GPU: NVIDIA Jetson AGX Orin 16GB video memory, memory: 32GB LPDDR5) have no large device dependencies and meet the requirements of lightweight deployment.

[0115] Software environment:

[0116] An experimental environment was built based on a deep learning framework, with supporting Transformer-related dependency libraries, to ensure the normal loading and computation of large visual language models.

[0117] Dataset: A standard training dataset for multimodal downstream tasks is selected, which contains k image-text pairs. A portion of the samples is partitioned into a cross-modal calibration sample set for gradient norm approximation calculation; simultaneously, a dataset shielding the visual modality is constructed according to the method of this invention. z is a dataset of noise and masked language modalities that are isomorphic to the image. , _ represents a meaningless null character.

[0118] In this embodiment, the selected dataset is the downstream task evaluation dataset of the BLIP and BLIP2 models to avoid mixing datasets across models, as shown in Table 1 below:

[0119] Table 1. Downstream task evaluation datasets for BLIP and BLIP2 models.

[0120]

[0121] All datasets were preprocessed in an "image-text pair" format, and when a single modality was masked, it was replaced with homomorphic noise (visual) or meaningless empty characters (linguistic).

[0122] The basic model uses a mainstream visual language model. The model weights include visual model weights and language model weights. The model has a visual encoder, a language encoder, and a cross-modal attention fusion module.

[0123] In this embodiment, only the XL specification model was tested, and the XXL specification was not tested; the language module uniformly uses T5, as detailed below:

[0124] Main experimental model: BLIP2-T5(xl) (visual module: ViT-L / 14, language module: T5-xl), with a total of approximately 3 billion parameters and an initial inference memory usage of 15.8GB (single sample batch=1).

[0125] Comparative experimental model: BLIP small model (visual module: ViT-B / 16, language module: T5-small), with a total of approximately 120 million parameters and an initial inference memory usage of 4.2GB (single sample batch=1), used only for the validation of the modal perception stage.

[0126] Core hyperparameter settings:

[0127] Noise sampling number K: Sets the number of noise samplings per sample, used for gradient norm approximation calculation, to ensure the stability of the gradient approximation;

[0128] Perturbation amplitude: Set the hyperparameter of the weight perturbation to control the degree of influence of noise on the weight;

[0129] Total parameter retention ratio: Preset the total parameter retention ratio of the model to clarify the model compression target;

[0130] Adaptive bias control parameters: Set the adaptive bias control parameters. The size parameter is used to fine-tune the modal share allocation.

[0131] In this embodiment, the core hyperparameters selected are:

[0132] Noise sampling number K=5 (to ensure the stability of the gradient norm approximation calculation);

[0133] Disturbance amplitude (Weight perturbation control to avoid excessive loss fluctuations);

[0134] The total parameter retention ratio is p=30% (target compression ratio, balancing model size and performance).

[0135] Adaptive bias control parameters (Modal share fine-tuning to adapt to different task dependencies);

[0136] Token pruning sparsity: retain the top 60% of valid tokens in the fusion score (filtering redundant cross-modal interaction information).

[0137] II. Pruning Procedure:

[0138] (I) Pruning procedure for the main experimental model (BLIP2-T5(xl)):

[0139] S1. Execute task awareness to determine downstream task modal dependencies:

[0140] Single-modal shielding and loss calculation:

[0141] Select d samples from the training dataset D. The loss of the model on the original data is calculated through forward propagation; similarly, the loss of the model on the original data is calculated through forward propagation. The loss on and in Losses;

[0142] According to the formula The loss increment of the masked visual modality is calculated to reflect the importance of the visual modality to downstream tasks.

[0143] According to the formula The loss increment of masked language modalities is calculated to reflect the importance of language modalities to downstream tasks.

[0144] Modal share allocation:

[0145] According to the formula Calculate the initial visual module share The square root operation is used to avoid the excessive proportion of a single mode in dual-modal driven tasks;

[0146] According to the formula Calculate adaptive bias and fine-tune modal shares to balance bimodal parameter retention;

[0147] According to the formula With formula To obtain the final visual share With text share This clarifies the dependence bias of downstream tasks on the two modes.

[0148] The share of each task mode is shown in Table 2 below.

[0149] Table 2 Task Modal Share

[0150]

[0151] S2. Perform the importance score calculation process:

[0152] Gradient norm approximation calculation:

[0153] Decompose the base model weights into visual model hierarchical weights. Hierarchical weights of language models ;

[0154] Regarding the first Layer target weight Visual layer Language layer According to the formula Calculate the approximate value of the gradient norm;

[0155] In the formula, That is, the noise values ​​follow a Gaussian distribution and are the same. The noise is reused when adding or subtracting noise. This represents the loss difference due to bidirectional disturbances.

[0156] The visual model was obtained respectively. Layer gradient norm With the language model Layer gradient norm .

[0157] Stratification importance score calculation:

[0158] According to the formula Computer vision module Layer importance score ;

[0159] According to the formula The computational language module Layer importance score ;

[0160] The sets of importance scores for the visual model and the language model are summarized to provide a basis for sparsity allocation.

[0161] S3. Execute the sparsity allocation process:

[0162] The importance scores of the visual model hierarchical division were normalized according to... Calculate the first Layer normalization ratio; similarly, according to Computational Language Model Layer normalization ratio;

[0163] In this embodiment, the sum of all scores in the visual layer is 45.2, and the normalized proportion of a certain layer is... The same applies to the language layer.

[0164] Calculate the total number of parameters retained. |W| represents the total number of parameters in the model;

[0165] According to the formula Computer vision model Layer sparsity , For the visual model The number of parameters in the layer;

[0166] According to the formula Computational Language Model Layer sparsity , For the language model The number of parameters in the layer;

[0167] Set a maximum sparsity threshold for each layer to prevent excessive pruning of parameters in a single-modal driven task, which could lead to functional failure.

[0168] S4. Perform a structured sparse process for token filtering:

[0169] Cross-modal token filtering:

[0170] In the cross-modal attention layer of the model, according to the formula and Calculate the attention weights of visual perception on text. Attention weights of text to vision ;

[0171] in, A query matrix for visual tokens. The query matrix for text tokens. A key matrix for visual tokens. The key matrix of text tokens. The attention head dimension is set to 64.

[0172] Attention weights for text Attention weights of text to vision By averaging across batches, multiple heads, and corresponding token dimensions, the text-token association degree of modal association degree for each token is obtained. and Visual token relevance;

[0173] According to the formula and Calculate the initial score of the visual token in a single modality. Initial score of text token unimodal Batch B is 16, and attention head H is 12.

[0174] According to the formula and The cross-modal fusion screening score is obtained;

[0175] Pruning sparsity according to a preset token sparseness, retaining a preset proportion of valid tokens before merging scores, forming a set of "key alignment points" for cross-modal interaction. and The first 60% of tokens are filtered to remove 40% of redundant cross-modal interaction information (such as meaningless background visual tokens and duplicate text tokens).

[0176] Structured pruning execution:

[0177] For each layer's weight matrix W, It is the in-layer weight matrix. It is the number of output channels. It is the number of input channels, according to the formula. Calculate the first Importance scores for each input channel;

[0178] in, For the first The average of the absolute values ​​of the elements of the input channel weight vector. For the corresponding number in the activation matrix The activation vectors of the input channels, The L2 norm of the activation vector;

[0179] Input channel Sort by size from largest to smallest to obtain the channel priority sequence. ;

[0180] Using a “2:4” structured sparse pattern, the sorted channels are divided into groups of four, with the top two most important channels in each group retained and the remaining two redundant channels removed.

[0181] Remove the columns corresponding to redundant channels in the weight matrix, skip the activation calculation of redundant channels during the inference phase, and complete the removal of redundant parameters within the layer.

[0182] (II) Comparative Experimental Model (BLIP Small Model) Experimental Procedure:

[0183] Only the S1 task perception step is executed. After determining the modal share of each downstream task, sparsity is directly allocated according to the modal share for simple channel pruning. The S2 importance score fine calculation and S4 token filtering steps are not executed. This is used to verify the independent effect of the modal perception step.

[0184] III. Verification of Experimental Results:

[0185] (a) Comparison of memory usage during pruning:

[0186] During the pruning process of the five downstream tasks, the peak memory usage was monitored in real time, and the comparison with the traditional gradient pruning method is shown in Table 3 below:

[0187] Table 3 Comparison of Peak Video Memory Usage

[0188]

[0189] The BLIP2-T5 (xl) uses only 4.4% of the memory of traditional methods. The memory usage of the BLIP small model (modal perception only) is further reduced, which greatly reduces the hardware resource requirements during the pruning stage.

[0190] (II) Performance comparison of the pruned models:

[0191] The task evaluation metrics of the BLIP and BLIP2 models (avoiding cross-model metric mixing) are used to compare the post-pruning performance of the original model before pruning, the post-pruning model of the present invention, and the BLIP mini-model (modality-aware only). The post-pruning performance is shown in Tables 4 and 5 below.

[0192] Table 4 Performance Comparison of BLIP2 Model

[0193]

[0194] Table 5 Performance Comparison of BLIP Small Models

[0195]

[0196] The average performance loss of the BLIP2 pruned model was only 2.3%, and the average performance loss of the BLIP mini-model (modal perception only) was 3.8%, both controlled within 4%, verifying the effectiveness of the method; moreover, the complete process (including token screening) can reduce performance loss more than modal perception only.

[0197] The clipped model BLIP2-T5(xl), with approximately 900 million parameters and a size of 34.2GB, was exported to ONNX format, quantized to FP16 precision using TensorRT, and then embedded into an NVIDIA Jetson AGX Orin embedded device to test the deployment effect.

[0198] Deployment feasibility: The model can be loaded and run directly without modifying hardware drivers or custom development, and the embedding process takes ≤10 minutes (content-based).

[0199] Inference performance: Single-sample inference latency (from input image-text to output result): COCO task ≤45ms, VQAV2 task ≤50ms, meeting real-time requirements (industrial-grade real-time standard ≤100ms).

[0200] Resource consumption: During inference, the video memory usage is ≤1.2GB, the CPU usage is ≤30%, and the memory usage is ≤8GB, which is fully adapted to the resource limitations of embedded devices.

[0201] Experimental results show that the task adaptation method of this invention achieves memory optimization on both BLIP and BLIP2 models, with pruning memory consumption being significantly lower than traditional methods, making it suitable for low-resource scenarios. The average performance loss of the pruned model is ≤3.8%, and the complete process (including token filtering) is better than modality perception alone, balancing lightweight design and task performance. The pruned model can be directly embedded into embedded devices, and its inference latency and resource consumption meet industrial-grade requirements, demonstrating practical application value.

[0202] Therefore, this invention provides a task-adaptive collaborative pruning method for large visual language models. It clarifies downstream task modal dependencies through task awareness and fine-tunes modal shares. Importance scores are calculated by combining gradients and modal losses, and sparsity is allocated to each layer according to normalized proportions. Redundant parameters are then pruned using a "2:4" structured sparsity combined with cross-modal tokens. This method effectively solves the problems of cross-modal degradation, poor task adaptation, and mis-pruning of key parameters in traditional pruning. While significantly compressing model size and improving inference efficiency, it accurately matches task modal requirements, avoiding performance loss. The pruned model structure is well-organized and can be directly accelerated using existing hardware, adapting to limited hardware resources. The pruned model structure remains intact, requiring no complex sparse storage format, making it easy to implement and deploy, and compatible with mainstream hardware deployments.

[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A task-adaptive collaborative pruning method for large visual language models, characterized in that, include: S1. Task awareness: Obtain the training dataset for multimodal downstream tasks, which contains multiple image-text pairs. By masking single-modal information in the training dataset, image-text pairs with masked visual information and image-text pairs with masked linguistic information are obtained respectively. The loss value when masking single-modal input is calculated using the above image-text pairs. Based on the loss value, the modal dependency of downstream tasks is determined. Adaptive bias is introduced to fine-tune the share allocation of the visual module and the linguistic module, and the visual share and text share corresponding to the downstream tasks are obtained. S2. Importance score calculation: The weights of the visual-language large model are divided into visual model weights and language model weights. The gradient norm approximation values ​​of the visual model layer and the language model layer are calculated using hierarchical weights. The gradient norm of each layer is multiplied by the share of the corresponding modality to obtain the importance scores of each layer of the visual module and each layer of the language module. In S2, the formula for calculating the approximate value of the gradient norm is: ； In the formula, For the first The target weights of the layer, where D is the cross-modal calibration sample set. To calibrate visual-text pairs in a cross-modal calibration sample set, The number of noise samples per sample. The loop index for the inner summation iterates from 1 to... All sampling rounds, For the disturbance amplitude to exceed the parameter, The difference in bidirectional disturbance loss during the j-th noise sampling; S3. Sparsity Allocation: Summarize the importance scores of the visual model and the language model, calculate the normalized proportion of the importance scores of each layer using a normalization method; based on the normalization proportion and the preset total parameter retention ratio, calculate the sparsity of each layer of the visual model and each layer of the language model, and set a maximum sparsity threshold for each layer. The formulas for calculating the sparsity of each layer are as follows: ； In the formula, For the visual model The sparsity of the layer, For the language model The sparsity of the layer, For the visual module Layer importance score, For the language module, layer 1 Importance score, For the corresponding mode number The normalized percentage of layer importance scores. , The preset percentage for retaining total parameters, where |W| represents the total number of parameters in the model. For the visual model The number of parameters in the layer For the language model The number of parameters in the layer; S4. Structured sparsity for token selection: Based on the pruning strategy of weight absolute value-activation value product, combined with the token selection mechanism guided by cross-modal attention weight, redundant parameters in each layer are pruned; firstly, the modal correlation degree of each token is calculated using bidirectional cross-modal attention weight, and the cross-modal fusion selection score is obtained by combining the initial single-modal score, and tokens with valid fusion scores are retained. The norm of the activated channel is calculated based on the valid tokens, and the importance score of the input channel is obtained by combining the absolute value of the weights. The input channels are sorted by importance score, and two channels are deleted in a 2:4 structured sparse pattern that retains the core parameter presets and removes redundant channels.

2. The task-adaptive collaborative pruning method for large visual language models according to claim 1, characterized in that, In S1, the specific formula for calculating the loss value when masking a single modal input is as follows: ； ； In the formula, Representing visual modality, Representing language modality, For cross-modal calibration sample set, To mask the dataset after the corresponding modality, This represents the mean loss value selected from the training dataset for each training sample. For the entire visual language model, This is the model loss function.

3. The task-adaptive collaborative pruning method for large visual language models according to claim 1, characterized in that, In S1, the formula for the initial allocation of visual module shares is: ； In the formula, For the initial visual module share, To mask the loss value when the visual modality is not displayed, The loss value when language modalities are masked; Introducing adaptive bias The formulas for calculating the visual share and text share are as follows: ； ； In the formula, For the final visual share, This is the final text share.

4. The task-adaptive collaborative pruning method for large visual language models according to claim 3, characterized in that, Adaptive bias The calculation formula is: ； In the formula, Parameters for controlling the size of the adaptive bias.

5. The task-adaptive collaborative pruning method for large visual language models according to claim 3, characterized in that, In S2, the formulas for calculating the importance scores of the visual model layer and the language model layer are as follows: ； In the formula, For the visual module Layer importance score, For the language module, layer 1 Importance score, For the visual model Layer weights, For the language model Layer weights, It is an L2 norm.

6. The task-adaptive collaborative pruning method for large visual language models according to claim 1, characterized in that, In S4, the formula for calculating the importance score of the input channel is: ； In the formula, It is the in-layer weight matrix. It is the number of output channels. This is the number of input channels. Indicates the first The absolute values ​​of the elements of the input channel weight vectors are taken and then averaged. For the first Importance scores for each input channel For the first The average of the absolute values ​​of the elements of the input channel weight vector. For the corresponding number in the activation matrix The activation vectors for each input channel, and the activation matrix has dimensions of . , , It refers to the batch size. Given the length of the input sequence, Total number of tokens For the corresponding number in the activation matrix The activation vectors of the input channels, For activation vectors Norm.

7. The task-adaptive collaborative pruning method for large visual language models according to claim 1, characterized in that, In S4, the formula for calculating the bidirectional cross-modal attention weights is: ； ； In the formula, The attention weight of the visual token over the text token. The attention weight of the text token on the visual token. A query matrix for visual tokens. The query matrix for text tokens. A key matrix for visual tokens. The key matrix of text tokens. The attention head dimension is denoted by 'attention head', and 'softmax' is the normalization function.

8. The task-adaptive collaborative pruning method for large visual language models according to claim 7, characterized in that, In S4, the formula for calculating the cross-modal fusion screening score is: ； ； In the formula, For cross-modal fusion of visual tokens, a score is selected. Filter scores for cross-modal fusion of text tokens. The initial score for the single-modal visual token. The initial score for the unimodal nature of the text token. The cross-modal attention weights are after batch averaging. The cross-modal attention weights are the result of averaging across multiple dimensions. The visual token's initial unimodal score Unimodal initial score of text token The calculation formula is: ； 。

Images