Multimodal large language model dynamic pruning method and system based on cross-modal activation perception and storage medium

Abstract

The application provides a multi-modal large language model dynamic pruning method and system based on cross-modal activation perception, and a storage medium. The method relies on a multi-field mixed calibration set and a modal feature decoupling mechanism to complete the separation statistics, logarithmic smoothing normalization and cross-modal fair fusion of text and visual heterogeneous features. Combined with an adaptive modal balance factor and a variable coefficient redundancy quantization, the method realizes precise weight importance evaluation and hierarchical sparsity dynamic allocation, and completes single weight unstructured pruning through mask and Hadamard product operation. The application avoids the defects of modal magnitude imbalance, inaccurate hierarchical sparse distribution, high computational overhead and generalization ability decay in traditional multi-modal pruning, improves the model performance retention rate and compression efficiency under high sparsity, guarantees the multi-modal alignment and reasoning ability integrity, and meets the deployment requirements of multi-modal large model lightweight, low delay and high performance on resource limited devices.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system and storage medium for dynamic pruning of multimodal large language models based on cross-modal activation perception. Background Technology

[0002] Multimodal large language models, by aligning deep features with a visual encoder, can achieve cross-modal tasks such as text-image understanding, graph reasoning, and document recognition. However, their parameter count can reach billions to tens of billions, resulting in massive memory consumption and computational overhead, making them difficult to deploy on resource-constrained devices. Model pruning is a mainstream lightweighting technique that aims to reduce model storage and computational costs by removing redundant parameters, channels, or network layers from neural networks.

[0003] Model pruning is mainly divided into structured pruning and unstructured pruning. Structured pruning compresses data by using channels and network layers as pruning units; unstructured pruning uses individual weights as pruning units, with algorithms such as Wanda and SparseGPT being typical examples in the pure text domain. The ECoFLaP hierarchical pruning framework for multimodal scenarios adopts a two-stage pruning strategy from coarse to fine. The coarse-grained stage evaluates the importance of network layers and assigns layer sparsity based on global zero-sample perplexity. The fine-grained stage calls traditional unstructured pruning algorithms to complete weight pruning.

[0004] However, structured pruning disrupts the Transformer network topology, leading to a sharp drop in multimodal alignment capabilities and requiring gradient backpropagation for retraining, resulting in extremely high computational costs. Unstructured pruning of pure text does not adapt to multimodal heterogeneous features, directly fusing textual and visual activation values, and masking key features with outliers. ECoFLaP cannot accurately quantify network layer redundancy, resulting in unbalanced sparsity distribution. At the same time, the calibration set has limited coverage of scenarios, and the pruning mask has distribution bias, significantly reducing the model's cross-domain generalization ability.

[0005] Therefore, it is necessary to propose a new model pruning method to solve the above-mentioned technical problems. Summary of the Invention

[0006] This invention provides a method, system, and storage medium for dynamic pruning of multimodal large language models based on cross-modal activation perception, aiming to solve the technical problem that model pruning is difficult to balance training cost and post-pruning model performance.

[0007] To address the aforementioned technical problems, in a first aspect, this invention provides a dynamic pruning method for multimodal large language models based on cross-modal activation perception, comprising the following steps:

[0008] S101. Determine the multimodal large language model to be pruned, and construct a multi-domain hybrid calibration dataset by fusing a pure text subset with a graph-text multimodal subset; S102. Input the multi-domain hybrid calibration set into the multimodal large language model to be pruned, decouple the input visual modal features and text modal features, and calculate the root mean square basic activation intensity of each channel in each network structure layer of the multimodal large language model to be pruned in the visual modal features and the text modal features respectively, so as to obtain the activation intensity of the visual modal channel and the activation intensity of the text modal channel respectively. S103. The visual modal channel activation intensity and the text modal channel activation intensity are respectively subjected to logarithmic smoothing mapping and infinite norm normalization to obtain normalized visual activation intensity and normalized text activation intensity. S104. The normalized visual activation intensity and the normalized text activation intensity are fused according to a preset adaptive modal balance factor to obtain the cross-modal balanced activation degree; at the same time, the original weight parameters of the multimodal large language model to be pruned are retrieved and their absolute values ​​are taken, and the weight importance score is calculated in combination with the cross-modal balanced activation degree. S105. Based on the cross-modal balanced activation degree, calculate the hierarchical variation coefficient of each network structure layer in the multimodal large language model to be pruned, and perform global standardization processing to output the standardized redundancy characterization value corresponding to each network structure layer. S106. Determine the preset global target sparsity. Based on the standardized redundancy characterization value, use a binary water-filling algorithm with a single-layer maximum sparsity safety upper limit and a single-layer minimum sparsity lower limit constraint, and combine the expansion coefficient to perform dynamic sparsity allocation to obtain the final sparsity of each of the network structure layers. S107. Generate a binary pruning mask based on the final sparsity and the weight importance score, and perform hierarchical adaptive unstructured pruning on the multimodal large language model to be pruned according to the binary pruning mask to obtain and output the pruned multimodal large language model.

[0009] Furthermore, in step S102, the activation intensity of the visual modality channel and the activation intensity of the text modality channel are defined as follows: , And satisfy the following relationship: ; ; in, The current network structure layer is a channel index. These are the token lengths for visual modality features and text modality features, respectively. For the first The first channel in the The activation value on each token.

[0010] Furthermore, in step S103, the normalized visual activation intensity and the normalized text activation intensity are defined as follows: , And satisfy the following relationship: ; ; Wherein, the normalized visual activation intensity The normalized text activation intensities are respectively Infinite norm After normalization, the values ​​are mapped to the interval [0, 1]. and These represent the global activation feature vectors formed by the original visual activation intensity and the original text activation intensity of all channels in the current network layer, respectively. This indicates the number of channels in the network layer. This represents the vector dimension of the network layer. For the preset minimum value smoothing term, The range of values ​​is .

[0011] Furthermore, in step S104, the cross-modal equilibrium activation degree is defined as... It satisfies the following relationship: ; in, The exponential decay function is driven by the preset global target sparsity. The weighted importance score is defined as follows: It satisfies the following relationship: ; in, To connect the input channel With output channel The original weight parameters.

[0012] Furthermore, in step S105, the standardized redundancy characterization value is defined as follows: It satisfies the following relationship: , ; in, Indicates the first The average value of the cross-modal balanced activation degree corresponding to all channels in each of the network structure layers, and , The range of values ​​is ; Indicates the first The standard deviation of the cross-modal balanced activations corresponding to all channels in the network structure layer, and , The range of values ​​is ; Indicates the first The layer variation coefficients of the network structure layers. This represents the average of the layer variation coefficients of all the aforementioned network structure layers. The standard deviation of the coefficient of variation of the layer for all the network structure layers is represented.

[0013] Furthermore, in step S106, the hierarchical sparsity function calculated based on the binary water injection algorithm is defined as follows: The global energy conservation constraint is The sparsity allocation strategy is They respectively satisfy the following relations: ; ; ; in, The preset global target sparsity, The total number of network layers in the multimodal large language model to be pruned. As the global baseline water level, For the preset expansion coefficient, The range of values ​​is , This represents the maximum sparsity safety limit for a single layer of the network structure. The minimum sparsity safety limit for a single layer of the network structure. The goal is to find the optimal global baseline water level by using binary search.

[0014] Furthermore, in step S107, the binarized pruning mask is defined as... It satisfies the following relationship: ; in, This represents a preset truncation threshold, which is determined by the weight importance score of the corresponding network structure layer. The larger the value, the more... , This indicates the number of weights ultimately retained in the network structure layer. This indicates the number of channels in the network structure layer; The conditions for defining hierarchical adaptive unstructured pruning are as follows: It satisfies the following relationship: ; in, Indicates the first The native weight parameters corresponding to all channels in the aforementioned network structure layer. It represents the Hadamardi (or Hadama) stack; when When the value is 0, the native weight parameters in the corresponding network structure layer are pruned.

[0015] Secondly, the present invention also provides a dynamic pruning system for multimodal large language models based on cross-modal activation perception, comprising: The dataset construction module is used to determine the multimodal large language model to be pruned, and to construct a multi-domain hybrid calibration dataset by fusing a pure text subset and a text-image multimodal subset; The decoupled activation module is used to input the multi-domain hybrid calibration set into the multimodal large language model to be pruned, decouple the input visual modal features and text modal features, and calculate the root mean square basic activation intensity of each channel in each network structure layer of the multimodal large language model to be pruned in the visual modal features and the text modal features respectively, so as to obtain the activation intensity of the visual modal channel and the activation intensity of the text modal channel respectively. The data mapping module is used to perform logarithmic smoothing mapping and infinite norm normalization on the visual modal channel activation intensity and the text modal channel activation intensity, respectively, to obtain normalized visual activation intensity and normalized text activation intensity. The modality fusion module is used to fuse the normalized visual activation intensity and the normalized text activation intensity according to a preset adaptive modality balance factor to obtain cross-modality balanced activation degree; at the same time, it retrieves the original weight parameters of the multimodal large language model to be pruned and takes the absolute value, and calculates the weight importance score in combination with the cross-modality balanced activation degree. The standard characterization module is used to calculate the hierarchical variation coefficient of each network structure layer in the multimodal large language model to be pruned based on the cross-modal balanced activation degree, and to perform global standardization processing to output the standardized redundancy characterization value corresponding to each network structure layer. The sparsity allocation module is used to determine the preset global target sparsity. Based on the standardized redundancy characterization value, it adopts a binary water-filling algorithm with a single-layer maximum sparsity safety upper limit and a single-layer minimum sparsity lower limit constraint, and combines the expansion coefficient to perform dynamic sparsity allocation at the execution level to obtain the final sparsity of each of the network structure layers. The pruning execution module is used to generate a binary pruning mask based on the final sparsity and the weight importance score, and to perform hierarchical adaptive unstructured pruning on the multimodal large language model to be pruned according to the binary pruning mask, so as to obtain and output the pruned multimodal large language model.

[0016] Thirdly, the present invention also provides a computer device, comprising: a memory, a processor, and a dynamic pruning program for a multimodal large language model based on cross-modal activation perception, stored in the memory and executable on the processor, wherein the dynamic pruning program for a multimodal large language model based on cross-modal activation perception implements the steps of the dynamic pruning method for a multimodal large language model based on cross-modal activation perception as described in any of the above embodiments.

[0017] Fourthly, the present invention also provides a storage medium storing a dynamic pruning program for a multimodal large language model based on cross-modal activation perception, wherein when the dynamic pruning program for a multimodal large language model based on cross-modal activation perception is executed by a processor, the program implements the steps of the dynamic pruning method for a multimodal large language model based on cross-modal activation perception as described in any of the above embodiments.

[0018] The beneficial effects achieved by this invention lie in proposing a dynamic pruning method for multimodal large language models based on cross-modal activation perception. This method relies on a multi-domain hybrid calibration set and a modal feature decoupling mechanism to complete the separation and statistical analysis of heterogeneous textual and visual features, logarithmic smoothing normalization, and cross-modal fair fusion. Combined with an adaptive modal balance factor and coefficient of variation redundancy quantization, it achieves accurate evaluation of weight importance and dynamic allocation of hierarchical sparsity. Single-weight unstructured pruning is completed through masking and Hadamard product operations. This invention avoids the shortcomings of traditional multimodal pruning, such as modal magnitude imbalance, inaccurate hierarchical sparsity allocation, excessive computational overhead, and attenuation of generalization ability. It improves the model performance retention rate and compression efficiency under high sparsity, ensures the integrity of multimodal alignment and inference capabilities, and meets the deployment requirements of lightweight, low-latency, and high-performance multimodal large models on resource-constrained devices. Attached Figure Description

[0019] The present invention will now be described in detail with reference to the accompanying drawings. The above and other aspects of the present invention will become clearer and more readily understood through the detailed description following the accompanying drawings. In the drawings: Figure 1 This is a flowchart illustrating the steps of the dynamic pruning method for multimodal large language models based on cross-modal activation perception provided in this embodiment of the invention. Figure 2 This is the execution logic diagram of the dynamic pruning method for multimodal large language models based on cross-modal activation perception provided in this embodiment of the invention; Figure 3This is a schematic diagram of the structure of the dynamic pruning system for multimodal large language models based on cross-modal activation perception provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0020] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] The specific embodiments / examples described herein are specific implementations of the present invention, used to illustrate the concept of the invention, and are illustrative and exemplary, and should not be construed as limiting the implementation methods or scope of the present invention. In addition to the embodiments described herein, those skilled in the art can employ other obvious technical solutions based on the content disclosed in the claims and specification of this application. These technical solutions include those that make any obvious substitutions and modifications to the embodiments described herein, all of which are within the protection scope of the present invention.

[0022] Example 1 Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a dynamic pruning method for multimodal large language models based on cross-modal activation perception, as provided in this embodiment of the invention. The dynamic pruning method for multimodal large language models based on cross-modal activation perception includes the following steps: S101. Determine the multimodal large language model to be pruned, and construct a multi-domain hybrid calibration dataset by fusing a pure text subset with a graph-text multimodal subset; S102. Input the multi-domain hybrid calibration set into the multimodal large language model to be pruned, decouple the input visual modal features and text modal features, and calculate the root mean square basic activation intensity of each channel in each network structure layer of the multimodal large language model to be pruned in the visual modal features and the text modal features respectively, so as to obtain the activation intensity of the visual modal channel and the activation intensity of the text modal channel respectively. S103. The visual modal channel activation intensity and the text modal channel activation intensity are respectively subjected to logarithmic smoothing mapping and infinite norm normalization to obtain normalized visual activation intensity and normalized text activation intensity. S104. The normalized visual activation intensity and the normalized text activation intensity are fused according to a preset adaptive modal balance factor to obtain the cross-modal balanced activation degree; at the same time, the original weight parameters of the multimodal large language model to be pruned are retrieved and their absolute values ​​are taken, and the weight importance score is calculated in combination with the cross-modal balanced activation degree. S105. Based on the cross-modal balanced activation degree, calculate the hierarchical variation coefficient of each network structure layer in the multimodal large language model to be pruned, and perform global standardization processing to output the standardized redundancy characterization value corresponding to each network structure layer. S106. Determine the preset global target sparsity. Based on the standardized redundancy characterization value, use a binary water-filling algorithm with a single-layer maximum sparsity safety upper limit and a single-layer minimum sparsity lower limit constraint, and combine the expansion coefficient to perform dynamic sparsity allocation to obtain the final sparsity of each of the network structure layers. S107. Generate a binary pruning mask based on the final sparsity and the weight importance score, and perform hierarchical adaptive unstructured pruning on the multimodal large language model to be pruned according to the binary pruning mask to obtain and output the pruned multimodal large language model.

[0023] Specifically, in this embodiment of the invention, step S101 aims to determine the target object and basic data support system for this pruning process. The selected multimodal large language model to be pruned integrates a visual encoder and a language model backbone, with a multi-layered neural network structure internally, possessing the ability to process text and images across modalities. The constructed multi-domain hybrid calibration dataset is composed of a plain text subset and a text-image multimodal subset. The plain text subset encompasses application scenarios such as general text, professional documents, and daily dialogues, while the text-image multimodal subset covers business types such as image recognition, chart parsing, document recognition, and mixed text-image recognition. In this embodiment of the invention, relying on diversified calibration samples, the feature activation performance of each level and channel of the model can be fully awakened, and the model's running state under real tasks can be completely collected, providing reliable data basis for subsequent feature decoupling, activation intensity calculation, and weight importance determination, ensuring that the model can still stably adapt to various cross-modal inference tasks after pruning.

[0024] In step S102, the activation intensity of the visual modality channel and the activation intensity of the text modality channel are defined as follows: , And satisfy the following relationship: ; ; in, The current network structure layer is a channel index. These are the token lengths for visual modality features and text modality features, respectively. For the first The first channel in the The activation value on each token.

[0025] Step S102 completes the modal feature decomposition and basic activation value calculation. First, after receiving the calibration dataset, the model distinguishes two independent modal features to avoid interference and superposition between different types of features. Then, using a single network structure layer as the processing unit, all channels within the layer are traversed layer by layer. The activation level of each channel is statistically calculated using the root mean square formula, which can objectively reflect the response degree of each channel in the process of modal information transmission. The activation intensity of the two types of modal channels is quantified by the formula, which can accurately characterize the contribution of different channels to the representation of visual and textual information.

[0026] In step S103, the normalized visual activation intensity and the normalized text activation intensity are defined as follows: , And satisfy the following relationship: ; ; Wherein, the normalized visual activation intensity The normalized text activation intensities are respectively Infinite norm After normalization, the values ​​are mapped to the interval [0, 1]. and These represent the global activation feature vectors formed by the original visual activation intensity and the original text activation intensity of all channels in the current network layer, respectively. This indicates the number of channels in the network layer. This represents the vector dimension of the network layer. For the preset minimum value smoothing term, The range of values ​​is .

[0027] Step S103 performs standardization correction on the activation intensity of the original channels in the dual-modal model. First, logarithmic smoothing is used to compress the range of activation values, weakening the interference caused by extreme outliers. Then, an infinite norm is introduced to complete global normalization, unifying the activation data of the two modalities to the same numerical range. The addition of a minimal smoothing term avoids the computational risk of a zero denominator, ensuring computational stability. This processing method can eliminate the representational bias caused by the difference in numerical magnitude between text and visual modalities, balance the contribution weights of the two types of information, and provide effective data with uniform scale for subsequent cross-modal activation fusion calculations.

[0028] In step S104, the cross-modal equilibrium activation degree is defined as... It satisfies the following relationship: ; in, The exponential decay function is driven by the preset global target sparsity. The weighted importance score is defined as follows: It satisfies the following relationship: ; in, To connect the input channel With output channel The original weight parameters.

[0029] Step S104 achieves dual-modal feature fusion and single-weight importance quantification evaluation. Relying on an adaptive modal balance factor, the proportion and weight of text and visual modalities are dynamically adjusted to generate balanced cross-modal activation levels that align with overall pruning and compression requirements, thus objectively reflecting the comprehensive information expression capability of a single channel. The score is calculated in conjunction with the absolute values ​​of the model's inherent weight parameters, taking into account both the parameter's own magnitude and the channel activation contribution, thereby reflecting the actual role of each weight in the model's feature transmission.

[0030] In step S105, the standardized redundancy characterization value is defined as follows: It satisfies the following relationship: , ; in, Indicates the first The average value of the cross-modal balanced activation degree corresponding to all channels in each of the network structure layers, and , The range of values ​​is ; Indicates the first The standard deviation of the cross-modal balanced activations corresponding to all channels in the network structure layer, and , The range of values ​​is ; Indicates the first The layer variation coefficients of the network structure layers. This represents the average of the layer variation coefficients of all the aforementioned network structure layers. The standard deviation of the coefficient of variation of the layer for all the network structure layers is represented.

[0031] Step S105 utilizes cross-modal balanced activation to quantify and standardize the redundant features of each network layer. Specifically, in this step, each layer is treated as an independent computational unit. Based on the cross-modal balanced activation of all channels within the layer, the mean and standard deviation of the activation are calculated separately. The layer variation coefficient is then calculated, which accurately reflects the dispersion and differential distribution of activation features within a single network layer. The more discrete the activation distribution, the stronger the effective representation capability of the layer and the lower the redundancy; conversely, the more discrete the distribution, the higher the layer redundancy. Based on this, global standardization is performed on the variation coefficients of all network layers to eliminate evaluation bias caused by differences in feature distribution between different network layers, outputting a standardized redundancy representation value with a unified metric.

[0032] The method in step S105 achieves an objective and quantitative assessment of the redundancy of the entire model, providing core quantitative indicators to support the accurate and reasonable dynamic allocation of sparsity to each network layer. This is the main design feature of the present invention that avoids the problems of ambiguous redundancy judgment and unbalanced sparsity allocation in traditional pruning methods.

[0033] In step S106, the hierarchical sparsity function calculated based on the binary water injection algorithm is defined as follows: The global energy conservation constraint is The sparsity allocation strategy is They respectively satisfy the following relations: ; ; ; in, The preset global target sparsity, The total number of network layers in the multimodal large language model to be pruned. As the global baseline water level, For the preset expansion coefficient, The range of values ​​is , This represents the maximum sparsity safety limit for a single layer of the network structure. The minimum sparsity safety limit for a single layer of the network structure. The goal is to find the optimal global baseline water level by using binary search.

[0034] In this embodiment of the invention, preferably, the maximum sparsity safety upper limit is... Minimum sparsity safety limit .

[0035] Step S106 aims to achieve adaptive and balanced dynamic allocation of sparsity across all network layers of the multimodal large language model.

[0036] Specifically, step S106 introduces a binary water injection algorithm to complete the layer sparsity allocation. This algorithm uses the global target sparsity as the total quota, regards the allocable sparsity of each layer as the water level space, and uses the benchmark water level line as a unified adjustment benchmark. Combined with the redundancy characterization value of each layer, the water level height of a single layer is adjusted in a differentiated manner with the expansion coefficient.

[0037] The binary water injection algorithm first defines the upper and lower limits of sparsity for a single layer to avoid over- or under-pruning of a single layer. Then, it iteratively calibrates the global baseline water level by relying on the binary search mechanism, continuously checks whether the average sparsity of all layers matches the global target sparsity, and continuously approaches the optimal water level value to finally satisfy the overall energy conservation constraint.

[0038] Using a globally preset target sparsity as a constraint benchmark, and combining the standardized redundancy representation values ​​of each layer, the algorithm allocates sparsity ratios under upper and lower limit constraints. Higher sparsity is allocated to layers with higher redundancy to increase pruning intensity, while the pruning magnitude is strictly controlled for core, low-redundancy layers to retain effective parameters. This algorithm abandons the traditional uniform fixed-ratio pruning mode, allocating pruning quotas differently based on the actual redundancy characteristics of each layer, and solving for the final sparsity of each network structure layer. This ensures both overall lightweight compression and maximizes the preservation of the model's cross-modal representation capabilities.

[0039] In step S107, the binarized pruning mask is defined as... It satisfies the following relationship: ; in, This represents a preset truncation threshold, which is determined by the weight importance score of the corresponding network structure layer. The larger the value, the more... , This indicates the number of weights ultimately retained in the network structure layer. This indicates the number of channels in the network structure layer; The conditions for defining hierarchical adaptive unstructured pruning are as follows: It satisfies the following relationship: ; in, Indicates the first The native weight parameters corresponding to all channels in the aforementioned network structure layer. It represents the Hadamardi (or Hadama) stack; when When the value is 0, the native weight parameters in the corresponding network structure layer are pruned.

[0040] Step S107 completes the final weight selection and pruning calculation based on the hierarchical pruning ratio and weight score. First, the number of weights to be retained at each layer is calculated based on the final sparsity, thereby determining the weight score truncation threshold. The retention and rejection ranges are then divided according to the single weight importance score, and a binary pruning mask containing only 0 and 1 values ​​is generated. The mask value can intuitively determine the weight retention status; a value of 1 represents retaining the corresponding weight, and a value of 0 represents discarding the corresponding weight.

[0041] Subsequently, a Hadamard product operation is performed between the original weight matrix of each layer and the pruning mask. Weight updates are achieved by multiplying corresponding positions, and weights with a result of zero are pruned. This method only sets individual weights to zero, while the overall network layer architecture and number of channels remain unchanged, representing a typical unstructured pruning approach. This embodiment of the invention utilizes layer-specific differentiated masks to achieve adaptive pruning, which not only meets the redundancy characteristics of each layer to achieve the predetermined compression goal but also maximizes the retention of high-contribution, effective weights, ultimately outputting a lightweight pruned model with stable inference performance.

[0042] Specifically, please refer to, for example Figure 2 The running logic diagram shown in this embodiment of the invention decouples, normalizes and fuses, and scores the modal features of the original model through steps S102 to S104. It separately calculates and then balances and fuses the dual-modal activations, accurately determines the parameter contribution from a single weight dimension, and thus determines the core basis for pruning evaluation. Step S105 measures the redundancy of each layer by using the coefficient of variation to evaluate the redundancy difference of each layer and distinguish between the core functional layer and the trimmable layer. Furthermore, in step S106, the bisection watering algorithm is used to dynamically allocate the sparsity of the layers, and the pruning ratio of each layer is determined by combining the upper and lower limit constraints to set a reasonable scale for layered pruning. Finally, in step S107, single-weight unstructured pruning is achieved using mask matching Hadamard product, thus realizing the final refined parameter reduction operation.

[0043] The pruning method proposed in this invention relies entirely on the forward inference statistical features of the calibration dataset, eliminating the need for additional iterative retraining and reverse fine-tuning. It only performs activation calculations, numerical statistics, and matrix operations, significantly reducing computational and time costs compared to existing methods. Simultaneously, it avoids feature representation imbalance through modal equalization; it allocates pruning amplitude differentially based on hierarchical redundancy, strictly controlling the upper limit of core layer pruning to protect key feature extraction capabilities; and it removes only low-contribution weights, retaining high-value parameters and the complete network and channel structure, effectively curbing accuracy loss and achieving lightweight compression while ensuring multimodal inference performance.

[0044] To demonstrate the effectiveness of the method proposed in this embodiment of the invention, the following experimental data is also provided. Specifically, in step S103, this embodiment proposes a method of processing activation intensity using logarithmic smoothing infinity norm normalization. For comparison, the output performance scores of the pruned model obtained after implementing step S103 with no normalization, maximum normalization, and quantile normalization, and completing the other steps proposed in this embodiment, are compared using ablation experiments. The ablation experiment uses the Qwen3-VL-8B-Instruct model as the model to be pruned, TextVQA as the test set, and iterates through the exponential decay function. of For the entire range of values, in the moderately sparse scenario, the overall pruning sparsity of the model is 50%, and in the extremely sparse scenario, the overall pruning sparsity of the model is 70%. The ablation experiment comparison results are shown in Table 1 below.

[0045] Table 1. Comparison of ablation experimental data for activation intensity treatment methods

[0046] It can be seen that under 50% sparsity, all normalization methods can maintain relatively reasonable performance, but log-smooth infinite norm normalization exhibits the highest ability to retain the upper limit and excellent hyperparameter stability. However, when the model is subjected to extreme sparsity of 70%, due to the sharp reduction in the capacity of the network structure layers, even small errors in cross-modal feature evaluation are amplified infinitely, leading to catastrophic performance collapse. Under this extreme condition, log-smooth infinite norm normalization shows a precipitous lead. Around λ=0.4, log-smooth infinite norm normalization still stubbornly retains an accuracy of 21.2 (far higher than the 8.2 of unnormalized and the 6.3 of quantile normalization), and also reaches 21.0 at λ=0.2. This demonstrates the effectiveness of the method of using log-smooth infinite norm normalization to handle activation intensity in improving the performance of the pruned model in the embodiments of this invention.

[0047] Furthermore, this embodiment of the invention also provides performance evaluation data comparing the model obtained by executing the pruning method proposed in this embodiment of the invention with the Qwen3-VL-8B-Instruct model as the model to be pruned, and comparing it with other existing pruning methods on five major benchmark test sets: TextVQA, ChartQA, OCR Bench, MME, and POPE. The comparison results are shown in Table 2 below.

[0048] Table 2 Performance comparison data between the pruned model and existing methods

[0049] As can be seen, at 50% sparsity, thanks to the cross-modal logarithmic smoothing and dynamic balancing mechanism, the proposed method (CMA_Log1P) in this embodiment of the invention exhibits global generalization performance that is highly close to that of the dense baseline model. At 65% sparsity, the global robustness and comprehensive advantages of the proposed method in this embodiment of the invention are demonstrated, showing a strong ability to retain multi-task balance under the impact of multi-dimensional chart data and complex layout.

[0050] The beneficial effects achieved by this invention lie in proposing a dynamic pruning method for multimodal large language models based on cross-modal activation perception. This method relies on a multi-domain hybrid calibration set and a modal feature decoupling mechanism to complete the separation and statistical analysis of heterogeneous textual and visual features, logarithmic smoothing normalization, and cross-modal fair fusion. Combined with an adaptive modal balance factor and coefficient of variation redundancy quantization, it achieves accurate evaluation of weight importance and dynamic allocation of hierarchical sparsity. Single-weight unstructured pruning is completed through masking and Hadamard product operations. This invention avoids the shortcomings of traditional multimodal pruning, such as modal magnitude imbalance, inaccurate hierarchical sparsity allocation, excessive computational overhead, and attenuation of generalization ability. It improves the model performance retention rate and compression efficiency under high sparsity, ensures the integrity of multimodal alignment and inference capabilities, and meets the deployment requirements of lightweight, low-latency, and high-performance multimodal large models on resource-constrained devices.

[0051] Example 2 This invention also provides a dynamic pruning system 200 for multimodal large language models based on cross-modal activation perception. Please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram of the structure of a dynamic pruning system for multimodal large language models based on cross-modal activation perception provided in an embodiment of the present invention, which includes: The dataset construction module 201 is used to determine the multimodal large language model to be pruned, and to construct a multi-domain hybrid calibration dataset by fusing a pure text subset and a text-image multimodal subset. The decoupled activation module 202 is used to input the multi-domain hybrid calibration set into the multimodal large language model to be pruned, decouple the input visual modal features and text modal features, and calculate the root mean square basic activation intensity of each channel in each network structure layer of the multimodal large language model to be pruned in the visual modal features and the text modal features respectively, so as to obtain the activation intensity of the visual modal channel and the activation intensity of the text modal channel respectively. Data mapping module 203 is used to perform logarithmic smoothing mapping and infinite norm normalization on the visual modal channel activation intensity and the text modal channel activation intensity respectively, so as to obtain normalized visual activation intensity and normalized text activation intensity respectively. The modality fusion module 204 is used to fuse the normalized visual activation intensity and the normalized text activation intensity according to a preset adaptive modality balance factor to obtain cross-modality balanced activation; at the same time, it retrieves the original weight parameters of the multimodal large language model to be pruned and takes the absolute value, and calculates the weight importance score in combination with the cross-modality balanced activation. The standard characterization module 205 is used to calculate the hierarchical variation coefficient of each network structure layer in the multimodal large language model to be pruned based on the cross-modal balanced activation degree, and to perform global standardization processing to output the standardized redundancy characterization value corresponding to each network structure layer. The sparsity allocation module 206 is used to determine the preset global target sparsity. Based on the standardized redundancy characterization value, it adopts a binary water-filling algorithm with a single-layer maximum sparsity safety upper limit and a single-layer minimum sparsity lower limit constraint, and combines the expansion coefficient to perform dynamic sparsity allocation at the execution level to obtain the final sparsity of each of the network structure layers. The pruning execution module 207 is used to generate a binary pruning mask based on the final sparsity and the weight importance score, and to perform hierarchical adaptive unstructured pruning on the multimodal large language model to be pruned according to the binary pruning mask, so as to obtain and output the pruned multimodal large language model.

[0052] The multimodal large language model dynamic pruning system 200 based on cross-modal activation perception can implement the steps in the multimodal large language model dynamic pruning method based on cross-modal activation perception in the above embodiments, and can achieve the same technical effect. Referring to the description in the above embodiments, it will not be repeated here.

[0053] Example 3 This invention also provides a computer device, please refer to... Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 300 includes: a memory 302, a processor 301, and a dynamic pruning program for a multimodal large language model based on cross-modal activation perception, which is stored in the memory 302 and can run on the processor 301.

[0054] The processor 301 calls the dynamic pruning program for a multimodal large language model based on cross-modal activation awareness stored in the memory 302, and executes the steps in the dynamic pruning method for a multimodal large language model based on cross-modal activation awareness provided in this embodiment of the invention. Please refer to... Figure 1 Specifically, it includes the following steps: S101. Determine the multimodal large language model to be pruned, and construct a multi-domain hybrid calibration dataset by fusing a pure text subset with a graph-text multimodal subset; S102. Input the multi-domain hybrid calibration set into the multimodal large language model to be pruned, decouple the input visual modal features and text modal features, and calculate the root mean square basic activation intensity of each channel in each network structure layer of the multimodal large language model to be pruned in the visual modal features and the text modal features respectively, so as to obtain the activation intensity of the visual modal channel and the activation intensity of the text modal channel respectively. S103. The visual modal channel activation intensity and the text modal channel activation intensity are respectively subjected to logarithmic smoothing mapping and infinite norm normalization to obtain normalized visual activation intensity and normalized text activation intensity. S104. The normalized visual activation intensity and the normalized text activation intensity are fused according to a preset adaptive modal balance factor to obtain the cross-modal balanced activation degree; at the same time, the original weight parameters of the multimodal large language model to be pruned are retrieved and their absolute values ​​are taken, and the weight importance score is calculated in combination with the cross-modal balanced activation degree. S105. Based on the cross-modal balanced activation degree, calculate the hierarchical variation coefficient of each network structure layer in the multimodal large language model to be pruned, and perform global standardization processing to output the standardized redundancy characterization value corresponding to each network structure layer. S106. Determine the preset global target sparsity. Based on the standardized redundancy characterization value, use a binary water-filling algorithm with a single-layer maximum sparsity safety upper limit and a single-layer minimum sparsity lower limit constraint, and combine the expansion coefficient to perform dynamic sparsity allocation to obtain the final sparsity of each of the network structure layers. S107. Generate a binary pruning mask based on the final sparsity and the weight importance score, and perform hierarchical adaptive unstructured pruning on the multimodal large language model to be pruned according to the binary pruning mask to obtain and output the pruned multimodal large language model.

[0055] Specifically, in step S102, the activation intensity of the visual modality channel and the activation intensity of the text modality channel are defined as follows: , And satisfy the following relationship: ; ; in, The current network structure layer is a channel index. These are the token lengths for visual modality features and text modality features, respectively. For the first The first channel in the The activation value on each token.

[0056] In step S103, the normalized visual activation intensity and the normalized text activation intensity are defined as follows: , And satisfy the following relationship: ; ; Wherein, the normalized visual activation intensity The normalized text activation intensities are respectively Infinite norm After normalization, the values ​​are mapped to the interval [0, 1]. and These represent the global activation feature vectors formed by the original visual activation intensity and the original text activation intensity of all channels in the current network layer, respectively. This indicates the number of channels in the network layer. This represents the vector dimension of the network layer. For the preset minimum value smoothing term, The range of values ​​is .

[0057] In step S104, the cross-modal equilibrium activation degree is defined as... It satisfies the following relationship: ; in, The exponential decay function is driven by the preset global target sparsity. The weighted importance score is defined as follows: It satisfies the following relationship: ; in, To connect the input channel With output channel The original weight parameters.

[0058] In step S105, the standardized redundancy characterization value is defined as follows: It satisfies the following relationship: , ; in, Indicates the first The average value of the cross-modal balanced activation degree corresponding to all channels in each of the network structure layers, and , The range of values ​​is ; Indicates the first The standard deviation of the cross-modal balanced activations corresponding to all channels in the network structure layer, and , The range of values ​​is ; Indicates the first The layer variation coefficients of the network structure layers. This represents the average of the layer variation coefficients of all the aforementioned network structure layers. The standard deviation of the coefficient of variation of the layer for all the network structure layers is represented.

[0059] In step S106, the hierarchical sparsity function calculated based on the binary water injection algorithm is defined as follows: The global energy conservation constraint is The sparsity allocation strategy is They respectively satisfy the following relations: ; ; ; in, The preset global target sparsity, The total number of network layers in the multimodal large language model to be pruned. As the global baseline water level, For the preset expansion coefficient, The range of values ​​is , This represents the maximum sparsity safety limit for a single layer of the network structure. The minimum sparsity safety limit for a single layer of the network structure. The goal is to find the optimal global baseline water level by using binary search.

[0060] In step S107, the binarized pruning mask is defined as... It satisfies the following relationship: ; in, This represents a preset truncation threshold, which is determined by the weight importance score of the corresponding network structure layer. The larger the value, the more... , This indicates the number of weights ultimately retained in the network structure layer. This indicates the number of channels in the network structure layer; The conditions for defining hierarchical adaptive unstructured pruning are as follows: It satisfies the following relationship: ; in, Indicates the first The native weight parameters corresponding to all channels in the aforementioned network structure layer. It represents the Hadamardi (or Hadama) stack; when When the value is 0, the native weight parameters in the corresponding network structure layer are pruned.

[0061] The computer device 300 provided in this embodiment of the invention can implement the steps in the dynamic pruning method for multimodal large language models based on cross-modal activation perception as described in the above embodiments, and can achieve the same technical effect. Referring to the description in the above embodiments, it will not be repeated here.

[0062] Example 4 This invention also provides a storage medium storing a dynamic pruning program for a multimodal large language model based on cross-modal activation perception. When executed by a processor, this dynamic pruning program implements the various processes and steps of the dynamic pruning method for a multimodal large language model based on cross-modal activation perception provided in this invention, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0063] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by hardware related to programs or instructions. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0064] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0065] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0066] The embodiments of the present invention have been described above with reference to the accompanying drawings. The disclosed embodiments are merely preferred embodiments of the present invention. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many equivalent changes in form under the guidance of the present invention without departing from the spirit and scope of the claims. All such changes are within the protection scope of the present invention.

Claims

1. A dynamic pruning method for multimodal large language models based on cross-modal activation perception, characterized in that, Includes the following steps: S101. Determine the multimodal large language model to be pruned, and construct a multi-domain hybrid calibration dataset by fusing a pure text subset with a graph-text multimodal subset; S102. Input the multi-domain hybrid calibration set into the multimodal large language model to be pruned, decouple the input visual modal features and text modal features, and calculate the root mean square basic activation intensity of each channel in each network structure layer of the multimodal large language model to be pruned in the visual modal features and the text modal features respectively, so as to obtain the activation intensity of the visual modal channel and the activation intensity of the text modal channel respectively. S103. The visual modal channel activation intensity and the text modal channel activation intensity are respectively subjected to logarithmic smoothing mapping and infinite norm normalization to obtain normalized visual activation intensity and normalized text activation intensity. S104. The normalized visual activation intensity and the normalized text activation intensity are fused according to a preset adaptive modal balance factor to obtain the cross-modal balanced activation degree; at the same time, the original weight parameters of the multimodal large language model to be pruned are retrieved and their absolute values ​​are taken, and the weight importance score is calculated in combination with the cross-modal balanced activation degree. S105. Based on the cross-modal balanced activation degree, calculate the hierarchical variation coefficient of each network structure layer in the multimodal large language model to be pruned, and perform global standardization processing to output the standardized redundancy characterization value corresponding to each network structure layer. S106. Determine the preset global target sparsity. Based on the standardized redundancy characterization value, use a binary water-filling algorithm with a single-layer maximum sparsity safety upper limit and a single-layer minimum sparsity lower limit constraint, and combine the expansion coefficient to perform dynamic sparsity allocation to obtain the final sparsity of each of the network structure layers. S107. Generate a binary pruning mask based on the final sparsity and the weight importance score, and perform hierarchical adaptive unstructured pruning on the multimodal large language model to be pruned according to the binary pruning mask to obtain and output the pruned multimodal large language model.

2. The dynamic pruning method for multimodal large language models based on cross-modal activation perception as described in claim 1, characterized in that, In step S102, the activation intensity of the visual modality channel and the activation intensity of the text modality channel are defined as follows: , And satisfy the following relationship: ； ； in, The current network structure layer is a channel index. These are the token lengths for visual modality features and text modality features, respectively. For the first The first channel in the The activation value on each token.

3. The dynamic pruning method for multimodal large language models based on cross-modal activation perception as described in claim 2, characterized in that, In step S103, the normalized visual activation intensity and the normalized text activation intensity are defined as follows: , And satisfy the following relationship: ； ； Wherein, the normalized visual activation intensity The normalized text activation intensities are respectively Infinite norm After normalization, the values ​​are mapped to the interval [0, 1]. and These represent the global activation feature vectors formed by the original visual activation intensity and the original text activation intensity of all channels in the current network layer, respectively. This indicates the number of channels in the network layer. This represents the vector dimension of the network layer. For the preset minimum value smoothing term, The range of values ​​is .

4. The dynamic pruning method for multimodal large language models based on cross-modal activation perception according to claim 3, characterized in that, In step S104, the cross-modal equilibrium activation degree is defined as... It satisfies the following relationship: ； in, The exponential decay function is driven by the preset global target sparsity. The weighted importance score is defined as follows: It satisfies the following relationship: ； in, To connect the input channel With output channel The original weight parameters.

5. The dynamic pruning method for multimodal large language models based on cross-modal activation perception according to claim 4, characterized in that, In step S105, the standardized redundancy characterization value is defined as follows: It satisfies the following relationship: ， ； in, Indicates the first The average value of the cross-modal balanced activation degree corresponding to all channels in each of the network structure layers, and , The range of values ​​is ; Indicates the first The standard deviation of the cross-modal balanced activations corresponding to all channels in the network structure layer, and , The range of values ​​is ; Indicates the first The layer variation coefficients of the network structure layers. This represents the average of the layer variation coefficients of all the aforementioned network structure layers. The standard deviation of the coefficient of variation of the layer for all the network structure layers is represented.

6. The dynamic pruning method for multimodal large language models based on cross-modal activation perception according to claim 5, characterized in that, In step S106, the hierarchical sparsity function calculated based on the binary water injection algorithm is defined as follows: The global energy conservation constraint is The sparsity allocation strategy is They respectively satisfy the following relations: ； ； ； in, The preset global target sparsity, The total number of network layers in the multimodal large language model to be pruned. As the global baseline water level, For the preset expansion coefficient, The range of values ​​is , This represents the maximum sparsity safety limit for a single layer of the network structure. The minimum sparsity safety limit for a single layer of the network structure. The goal is to find the optimal global baseline water level by using binary search.

7. The dynamic pruning method for multimodal large language models based on cross-modal activation perception according to claim 6, characterized in that, In step S107, the binarized pruning mask is defined as... It satisfies the following relationship: ； in, This represents a preset truncation threshold, which is determined by the weight importance score of the corresponding network structure layer. The larger the value, the more... , This indicates the number of weights ultimately retained in the network structure layer. This indicates the number of channels in the network structure layer; The conditions for defining hierarchical adaptive unstructured pruning are as follows: It satisfies the following relationship: ； in, Indicates the first The native weight parameters corresponding to all channels in the aforementioned network structure layer. It represents the Hadamardi (or Hadama) stack; when When the value is 0, the native weight parameters in the corresponding network structure layer are pruned.

8. A dynamic pruning system for multimodal large language models based on cross-modal activation perception, characterized in that, include: The dataset construction module is used to determine the multimodal large language model to be pruned, and to construct a multi-domain hybrid calibration dataset by fusing a pure text subset and a text-image multimodal subset; The decoupled activation module is used to input the multi-domain hybrid calibration set into the multimodal large language model to be pruned, decouple the input visual modal features and text modal features, and calculate the root mean square basic activation intensity of each channel in each network structure layer of the multimodal large language model to be pruned in the visual modal features and the text modal features respectively, so as to obtain the activation intensity of the visual modal channel and the activation intensity of the text modal channel respectively. The data mapping module is used to perform logarithmic smoothing mapping and infinite norm normalization on the visual modal channel activation intensity and the text modal channel activation intensity, respectively, to obtain normalized visual activation intensity and normalized text activation intensity. The modality fusion module is used to fuse the normalized visual activation intensity and the normalized text activation intensity according to a preset adaptive modality balance factor to obtain cross-modality balanced activation degree; at the same time, it retrieves the original weight parameters of the multimodal large language model to be pruned and takes the absolute value, and calculates the weight importance score in combination with the cross-modality balanced activation degree. The standard characterization module is used to calculate the hierarchical variation coefficient of each network structure layer in the multimodal large language model to be pruned based on the cross-modal balanced activation degree, and to perform global standardization processing to output the standardized redundancy characterization value corresponding to each network structure layer. The sparsity allocation module is used to determine the preset global target sparsity. Based on the standardized redundancy characterization value, it adopts a binary water-filling algorithm with a single-layer maximum sparsity safety upper limit and a single-layer minimum sparsity lower limit constraint, and combines the expansion coefficient to perform dynamic sparsity allocation at the execution level to obtain the final sparsity of each of the network structure layers. The pruning execution module is used to generate a binary pruning mask based on the final sparsity and the weight importance score, and to perform hierarchical adaptive unstructured pruning on the multimodal large language model to be pruned according to the binary pruning mask, so as to obtain and output the pruned multimodal large language model.

9. A computer device, characterized in that, include: The system includes a memory, a processor, and a dynamic pruning program for a multimodal large language model based on cross-modal activation awareness, which is stored in the memory and can run on the processor. When the processor executes the dynamic pruning program for a multimodal large language model based on cross-modal activation awareness, it implements the steps in the dynamic pruning method for a multimodal large language model based on cross-modal activation awareness as described in any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores a dynamic pruning program for a multimodal large language model based on cross-modal activation perception. When the processor executes the dynamic pruning program for a multimodal large language model based on cross-modal activation perception, it implements the steps in the dynamic pruning method for a multimodal large language model based on cross-modal activation perception as described in any one of claims 1-7.

Images