Multi-modal large model quantification method and system based on visual marker importance weighting

Abstract

The invention provides a multi-modal large model quantification method and system based on visual marker importance weighting, and relates to the technical field of computer vision. The method comprises the following steps: inputting an obtained vision-language sample set into a trained multi-modal large model, on the premise of not retraining the model, performing one-time back propagation on multi-modal interaction layer marking representation by using a small number of vision-language calibration samples, calculating gradient norms mark by mark, and performing multi-modal interaction layer marking representation. Dividing an important visual mark and a redundant visual mark, counting an average gradient together with the text mark, and normalizing to obtain a weight coefficient; different mark errors are weighted according to weights in quantization reconstruction loss, so that quantization precision is concentrated on core information, quantization errors of key visual marks are remarkably reduced, and precision and robustness of a multi-modal large model after quantization in tasks such as visual questions and answers and image-text dialogues are improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for quantizing multimodal large models based on visual label importance weighting. Background Technology

[0002] With the development of large-scale pre-training techniques, large-scale vision-language models, through visual encoding sub-networks, text encoding sub-networks, and multimodal interaction sub-networks, can perform various tasks such as visual question answering, image-text dialogue, and image description within a unified framework. However, such models typically contain billions of parameters and require processing a large number of visual and text tags during the inference phase, resulting in high memory consumption and computational overhead, severely limiting their deployment in scenarios such as general-purpose GPUs and edge inference devices. Post-training quantization is a technique that can compress models without retraining and relies only on a small number of calibration samples. For large language models, various post-training quantization methods have maintained relatively acceptable performance at lower bit widths through strategies such as linear uniform quantization, intra-group reconstruction, and dynamic thresholding. However, directly applying these methods to large-scale vision-language models often ignores the different roles of visual and text tags in the task.

[0003] Some existing quantization methods for multimodal models introduce modality-level weights into the loss function, assigning different weights to the overall quantization errors of the visual and text modalities to balance the contributions of different modalities to the final task metrics. However, within the same visual modality, all visual markers are still assumed to have the same importance, and the quantization error is evenly distributed across critical and redundant regions.

[0004] In practice, to cover a sufficient amount of spatial locations and details, large-scale vision-language models often involve densely segmenting images or sampling multiple frames of video. Experiments and experience show that only a portion of the visual tags play a decisive role in the answer or generated content, while the remaining tags are highly semantically repetitive or have almost no impact on the output. On the other hand, the parameter scale of multimodal large-scale models is extremely large. Introducing complete retraining or complex fine-tuning during the quantization stage would result in enormous computational overhead, contradicting the original intention of post-training quantization to be "lightweight and fast."

[0005] Therefore, there is a need for an importance estimation mechanism that can be completed without updating model parameters and with only a small number of calibration samples, and can automatically distinguish between key and redundant information inside visual labels during the quantization process, so as to allocate limited quantization precision more effectively. Summary of the Invention

[0006] The present invention provides a multimodal large model quantization method and system based on visual label importance weighting, which aims to significantly reduce the performance degradation caused by quantization while maintaining the simplicity and efficiency of the post-training quantization process.

[0007] Firstly, the multimodal large model quantization method based on visual marker importance weighting provided by this invention mainly includes: Obtain a visual-language sample set for quantization calibration, input the visual-language sample set into a pre-trained multimodal large model, and obtain the output of the multimodal large model and the corresponding loss value; Without updating the parameters of the multimodal large model, the loss value is backpropagated once relative to the intermediate label representation of the layer to be quantized in the multimodal large model, and the gradient information is calculated for each label to obtain the gradient importance measure of each visual label and text label in each sample of the visual-language sample set. For each sample, visual markers are sorted according to gradient importance metric, and visual markers whose gradient importance metric is within the top preset percentage range are selected as important visual markers, while the remaining visual markers are considered redundant visual markers. The gradient importance measures of important visual markers, redundant visual markers and text markers are averaged and statistically analyzed to obtain the average gradient of important visual markers, the average gradient of redundant visual markers and the average gradient of text markers. Then, they are normalized to obtain the first weight coefficient, the second weight coefficient and the third weight coefficient, respectively. In the post-training quantization process of the multimodal large model, with weighted reconstruction error as the optimization objective, the error related to important visual markers is multiplied by the first weight coefficient, the error related to redundant visual markers is multiplied by the second weight coefficient, and the error related to text markers is multiplied by the third weight coefficient. This is achieved by optimizing the scaling matrix. The value of minimizes the weighted reconstruction error, and the model weights and activations are quantized with low bits. The multimodal large model, after post-training quantization based on the importance of visual markers, is used to perform visual-language task inference.

[0008] Furthermore, the step-by-step gradient information calculation includes: Loss function for a given calibration sample Based on this, for the first The first sample The hidden representation vector corresponding to each tag Calculate the gradient norm: ; in, express norm, the gradient norm For marking In the sample Important indicators in the process.

[0009] Further, the visually marked ones with gradient importance measures in the top 30% are selected as important visual markings, including: Let the total number of visually marked ones in the th sample be , and sort the visually marked ones in this sample in descending order, and take the first visually marked ones to form the set of important visual markings , and the remaining visually marked ones form the set of redundant visual markings , where represents the ceiling operation.

[0010] Further, the average statistics of the gradient importance measures of the important visual markings, redundant visual markings, and text markings are respectively performed to obtain the average gradient of important visual markings, the average gradient of redundant visual markings, and the average gradient of text markings, including: Let the important visual markings, redundant visual markings, and average gradient of text markings of the th sample be respectively represented as: ; ; ; where is the set of text markings in the th sample.

[0011] Further, the normalization is performed to respectively obtain the first weight coefficient, the second weight coefficient, and the third weight coefficient, including: Let the weight coefficients of the important visual markings, redundant visual markings, and text markings be respectively represented as: ; ; ; and satisfy: .

[0012] Further, the triggering of the cross - level linkage response mechanism includes: Switch to the backup communication channel; Lock the access permission of the suspicious device; Notify the security management personnel to intervene in the handling, and the notification is based on the security event bus event.

[0013] Further, the low - bit quantization of the model weights and activations includes: For any input floating-point value The linear uniform quantization operator Represented as: ; Where s is the quantization step size and z is the zero offset. This indicates the rounding operation; and These are the minimum and maximum integer values ​​determined by the quantization bit width; The truncation function is defined as follows: .

[0014] Furthermore, the method includes: For signed Bit quantization, selected: .

[0015] Furthermore, the method includes: Given a tensor to be quantized Under the conditions, and Calculate the minimum and maximum values ​​of the tensor to be quantized according to the following preset rules:

[0016] Then let ; in, This indicates the rounding operation.

[0017] Furthermore, the method includes: For a certain linear layer weight tensor In the scaling matrix Under the influence of [the specific function], the weighted reconstruction error objective function of the linear layer is expressed as:

[0018]

[0019]

[0020] in, , and The first The important visual marker input matrix, redundant visual marker input matrix, and text marker input matrix of each sample. express Norm.

[0021] Secondly, the multimodal large model quantization system based on visual marker importance weighting provided by this invention mainly includes: The acquisition module is used to acquire a visual-language sample set for quantization calibration, input the visual-language sample set into a trained multimodal large model, and obtain the output of the multimodal large model and the corresponding loss value. The gradient metric module is used to perform a backpropagation of the loss value relative to the intermediate label representation of the layer to be quantized in the multimodal large model without updating the parameters of the multimodal large model, calculate the gradient information for each label, and obtain the gradient importance metric of each visual label and text label in each sample of the visual-language sample set. The sorting and selection module is used to sort the visual markers according to the gradient importance metric for each sample, and select the visual markers whose gradient importance metric is within the top preset percentage range as important visual markers, and the remaining visual markers as redundant visual markers. The statistics module is used to perform average statistics on the gradient importance measures of important visual markers, redundant visual markers and text markers respectively, to obtain the average gradient of important visual markers, the average gradient of redundant visual markers and the average gradient of text markers, and then normalize them to obtain the first weight coefficient, the second weight coefficient and the third weight coefficient respectively. The bit quantization module is used, during the post-training quantization process of the multimodal large model, to optimize the weighted reconstruction error by multiplying the error related to important visual markers by the first weight coefficient, the error related to redundant visual markers by the second weight coefficient, and the error related to text markers by the third weight coefficient, and by optimizing the scaling matrix. The value of minimizes the weighted reconstruction error, and the model weights and activations are quantized with low bits. The execution inference module is used to perform visual-language task inference using the multimodal large model after post-training quantization processing based on the importance of visual tags.

[0022] The technical solutions provided by the embodiments of the present invention have the following beneficial effects: 1. Label-by-label importance modeling: This invention evaluates label importance based on label-by-label gradient norm, and divides visual labels into important and redundant categories within each sample, which significantly refines the characterization of visual modalities.

[0023] 2. Sample-level adaptive weighting: Each calibration sample independently calculates the three-class average gradients and normalizes them to obtain weights, so that different images have different weight coefficients and important and redundant label distributions, which can adaptively emphasize the key regions of each sample.

[0024] 3. Quantization accuracy is focused on core information: Since the average gradient of important visual markers is large, their error has a higher weight in the objective function. The optimization process naturally tends to reduce this part of the error, thereby "focusing quantization accuracy on core information" and effectively mitigating damage to key visual areas.

[0025] 4. No need to retrain the model: This invention does not update the original model parameters throughout the entire process. It only uses a small amount of calibration data to perform backpropagation and scaling matrix optimization once, which has low computational overhead and is easy to deploy in actual engineering. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall process of the multimodal large model quantization method based on visual label importance weighting provided in an embodiment of the present invention.

[0027] Figure 2 Applications provided in the embodiments of the present invention Figure 1 A schematic diagram of the weighted quantization loss construction.

[0028] Figure 3 Applications provided in the embodiments of the present invention Figure 2 A comparison chart of inference results on the digit recognition task. Detailed Implementation

[0029] To further understand the content of this invention, a detailed description of the invention is provided in conjunction with the accompanying drawings and embodiments. The specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0030] Example 1

[0031] like Figures 1-3 As shown, the multimodal large model quantization method based on visual label importance weighting provided in this embodiment of the invention may specifically include the following steps: Step S100: Obtain a visual-language sample set for quantization calibration, input the visual-language sample set into the trained multimodal large model, and obtain the output of the multimodal large model and the corresponding loss value.

[0032] In this embodiment, the initial conditions are as follows: Pre-trained multimodal large model It includes a visual encoding subnetwork, a text encoding subnetwork, a multimodal interaction subnetwork, and a decoding subnetwork. Calibration sample set. , in For the first A sample of images or video frames, For text sequences. Target quantization bit width. Set of layers to be quantized For example, linear layers in multimodal interaction layers and decoding layers.

[0033] Forward reasoning and loss calculation: For each calibration sample In the full-precision model The task is performed by forward inference, and the loss function is calculated according to the preset task. .

[0034] Step S200: Without updating the parameters of the multimodal large model, perform a backpropagation on the loss value relative to the intermediate label representation of the layer to be quantized in the multimodal large model, calculate the gradient information for each label, and obtain the gradient importance measure of each visual label and text label in each sample of the visual-language sample set.

[0035] When calculating the gradient for each label, all model parameters are kept constant, and the loss function is... A backpropagation is performed relative to the label representation of the intermediate layer of the multimodal interaction subnetwork and the decoding subnetwork.

[0036] Suppose that on the selected marker layer, the first... The first sample The hidden representation of each marker is as follows: Then the gradient importance metric for this label is defined as: ; in: for Norm. Through the above calculations, the first norm can be obtained. The set of gradient importance of all visual and text tags in the sample. .

[0037] This embodiment evaluates label importance based on label-by-label gradient norm, classifying visual labels into important and redundant categories within each sample, significantly refining the characterization of visual modalities. Throughout the process, the original model parameters are not updated; only a small amount of calibration data is used for backpropagation and scaling matrix optimization, resulting in low computational overhead and ease of deployment in practical engineering.

[0038] Step S300: For each sample, sort the visual markers according to the gradient importance metric, select the visual markers whose gradient importance metric is within the top preset percentage range as important visual markers, and the remaining visual markers as redundant visual markers.

[0039] For the first Sample, let This is a set of visual tag indexes, with a total of 100 visual tags. According to gradient importance value Sort by size from largest to smallest, and take the first one. Visual markers constitute an important set of visual markers. The remaining visual markers form a redundant set of visual markers. in, This represents the floor operation, which rounds up to the smallest integer not less than a given real number. Through this step, each sample obtains its own specific distribution of visual label importance and redundancy.

[0040] Step S400: Perform average statistics on the gradient importance measures of important visual markers, redundant visual markers and text markers respectively to obtain the average gradient of important visual markers, the average gradient of redundant visual markers and the average gradient of text markers, and normalize them to obtain the first weight coefficient, the second weight coefficient and the third weight coefficient respectively.

[0041] make For the first The set of text tag indexes for the sample, for the first The average gradient of the three classes of labels in the sample was calculated separately, and the important visual labels were... Redundant visual markers and average gradient of text tags They are respectively:

[0042]

[0043]

[0044] Then normalization is performed to obtain the first... The three weighting coefficients of each sample: , , , satisfy: ; In this way, each sample has an independent set of important visual label weights, redundant visual label weights, and text label weights.

[0045] In this embodiment, three types of average gradients are independently calculated for each calibration sample and normalized to obtain weights, so that different images have different weight coefficients and important and redundant label distributions, which can adaptively emphasize the key regions of each sample.

[0046] Step S500: During the post-training quantization process of the multimodal large model, with the weighted reconstruction error as the optimization objective, the error related to important visual markers is multiplied by the first weight coefficient, the error related to redundant visual markers is multiplied by the second weight coefficient, and the error related to text markers is multiplied by the third weight coefficient. This is achieved by optimizing the scaling matrix. The value of minimizes the weighted reconstruction error, and low-bit quantization is performed on the model weights and activations.

[0047] In this embodiment, a linear uniform quantization operator can be used for the weight tensor or activation tensor that needs to be quantized. For any floating-point value Its quantitative form is:

[0048] in, To quantize the step size, Zero-point offset, This indicates the rounding operation. For quantization bit width The minimum and maximum integer values ​​to determine, for example, for signed integers. Bit quantization is acceptable This indicates the rounding operation; The truncation operation is defined as follows:

[0049] Quantization step size and zero point The following preset rules are used to determine the tensor to be quantized. First, find its minimum and maximum values:

[0050] Then let:

[0051] This embodiment does not and Instead of performing additional optimizations, it is treated as a parameter uniquely determined by the rules described above.

[0052] Since the average gradient of important visual markers is large, their error has a higher weight in the objective function. The optimization process naturally tends to reduce this part of the error, thereby "concentrating the quantization accuracy on the core information" and effectively mitigating the damage to key visual regions.

[0053] Step S600: Perform visual-language task inference using the multimodal large model after post-training quantization processing based on visual tag importance weighting.

[0054] For a given linear layer to be quantized, its full-precision weight tensor is denoted as . This embodiment introduces a diagonal scaling matrix. Transform the calculation of this layer into a weighted side: Input side: .right Quantify to obtain For the first For each sample, extract the submatrices corresponding to important visual markers, redundant visual markers, and text markers from the input tensor of this layer, denoted as follows:

[0055] The corresponding full-precision output and quantization output are respectively

[0056]

[0057]

[0058] choose Using the norm as a measure of reconstruction error, the weighted reconstruction objective in this embodiment can be written as:

[0059]

[0060]

[0061] In practical implementation, it can be done in the following ways. Optimize: set up It is a diagonal matrix, with diagonal elements The scalar to be optimized; the above summation is approximated using a small batch of calibration samples. The grid search algorithm is used to iteratively update the algorithm several times; during the optimization process, the algorithm is maintained. No change, only update , , Each time according to the new Recalculate according to the preset rules. When the preset number of iterations is reached, the current... Consider as the optimal scaling matrix .

[0062] Model Quantization and Deployment: Repeat steps S500 and S600 for all layers to be quantized to obtain the optimal scaling matrix and quantization weights for each layer. Replace the floating-point weights in the original model with quantized weights, and adjust accordingly if necessary. The model is folded together with the previous or subsequent operators to form the final quantized model. During the inference phase, integer operators and pre-defined quantization inverse calculation rules are used for forward computation, enabling efficient execution of vision-language tasks on general-purpose servers or edge devices.

[0063] Example 2

[0064] Based on Example 1, this example provides an implementation method to improve the stability of label importance estimation by utilizing multi-layer gradient information, which is used to estimate the label importance of multi-layer information fusion.

[0065] In step S200, gradients are calculated not only in the last layer of the multimodal interactive subnetwork, but also in several intermediate layers. Let the selected set of layer indices be... , No. The first layer Sample No. The hidden representation of each marker is as follows: The gradient norm is:

[0066] The importance of the comprehensive gradient can then be defined as...

[0067] in The weighting coefficients are non-negative and can be the mean weights. Different weights can also be set according to layer position. The subsequent steps are the same as in Example 1; simply add the new... Simply substitute the values. By using multi-layer fusion, the impact of single-layer gradient noise on importance ranking can be reduced, thus improving the robustness of important visual marker recognition.

[0068] Example 3: Precision Protection Strategy Combining Bit Width Allocation

[0069] Based on Embodiment 1 and Embodiment 2, this embodiment further provides an implementation method that combines bit width allocation to enhance the accuracy protection of important visual areas.

[0070] Specifically, after optimizing the scaling matrix and weighted reconstruction quantization for each layer, based on the distribution of important visual markers across different layers of the network, higher bit widths are used for some layers, while lower bit widths are used for the rest. For example, the contribution of visual markers with the largest gradients in each layer is statistically analyzed. If a layer frequently produces visual markers with high gradient values ​​in most samples, it is marked as a "critical layer." The weights of critical layers are quantized using 8 bits, while those of non-critical layers are quantized using 4 bits.

[0071] This embodiment allocates more effective quantization precision to layers sensitive to important visual markers without significantly increasing the overall model size and inference time, thereby further improving the overall performance of the low-bit quantization model.

[0072] Example 4

[0073] This embodiment provides a multimodal large model quantization system based on visual label importance weighting to implement the above method. The system includes: an acquisition module, a gradient measurement module, a sorting and selection module, a statistics module, a bit quantization module, and an execution inference module. Wherein: The acquisition module is used to acquire a visual-language sample set for quantization calibration, input the visual-language sample set into a pre-trained multimodal large model, and obtain the output of the multimodal large model and the corresponding loss value.

[0074] The gradient metric module is used to perform a backpropagation of the loss value relative to the intermediate label representation of the layer to be quantized in the multimodal large model without updating the parameters of the multimodal large model, calculate the gradient information for each label, and obtain the gradient importance metric of each visual label and text label in each sample of the visual-language sample set.

[0075] The sorting and selection module is used to sort the visual markers according to the gradient importance metric for each sample, select the visual markers whose gradient importance metric is within the top preset percentage range as important visual markers, and the remaining visual markers as redundant visual markers.

[0076] The statistical module is used to perform average statistics on the gradient importance measures of important visual markers, redundant visual markers and text markers respectively, to obtain the average gradient of important visual markers, the average gradient of redundant visual markers and the average gradient of text markers, and to normalize them to obtain the first weight coefficient, the second weight coefficient and the third weight coefficient respectively.

[0077] The bit quantization module is used, during the post-training quantization process of the multimodal large model, to optimize the weighted reconstruction error by multiplying the error related to important visual markers by the first weight coefficient, the error related to redundant visual markers by the second weight coefficient, and the error related to text markers by the third weight coefficient, thereby optimizing the scaling matrix. The value of minimizes the weighted reconstruction error, and low-bit quantization is performed on the model weights and activations.

[0078] The execution inference module is used to perform visual-language task inference using the multimodal large model after post-training quantization processing based on visual tag importance weighting.

[0079] It should be noted that the modules or units provided in the embodiments of the present invention have the same implementation principle and technical effects as those in the aforementioned method embodiments. For the sake of brevity, the specific working process of the modules described above can be referred to the corresponding process in the aforementioned method embodiments, and will not be repeated here.

[0080] In summary, the technical solutions provided by the embodiments of the present invention have the following beneficial effects: 1. Label-by-label importance modeling: This invention evaluates label importance based on label-by-label gradient norm, and divides visual labels into important and redundant categories within each sample, which significantly refines the characterization of visual modalities.

[0081] 2. Sample-level adaptive weighting: Each calibration sample independently calculates the three-class average gradients and normalizes them to obtain weights, so that different images have different weight coefficients and important and redundant label distributions, which can adaptively emphasize the key regions of each sample.

[0082] 3. Quantization accuracy is focused on core information: Since the average gradient of important visual markers is large, their error has a higher weight in the objective function. The optimization process naturally tends to reduce this part of the error, thereby "focusing quantization accuracy on core information" and effectively mitigating damage to key visual areas.

[0083] 4. No need to retrain the model: This invention does not update the original model parameters throughout the entire process. It only uses a small amount of calibration data to perform backpropagation and scaling matrix optimization once, which has low computational overhead and is easy to deploy in actual engineering.

[0084] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0085] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal large model quantization method based on visual tag importance weighting, characterized by: include: Obtain a visual-language sample set for quantization calibration, input the visual-language sample set into a pre-trained multimodal large model, and obtain the output of the multimodal large model and the corresponding loss value; Without updating the parameters of the multimodal large model, the loss value is backpropagated once relative to the intermediate label representation of the layer to be quantized in the multimodal large model, and the gradient information is calculated for each label to obtain the gradient importance measure of each visual label and text label in each sample of the visual-language sample set. For each sample, visual markers are sorted according to gradient importance metric, and visual markers whose gradient importance metric is within the top preset percentage range are selected as important visual markers, while the remaining visual markers are considered redundant visual markers. The gradient importance measures of important visual markers, redundant visual markers and text markers are averaged and statistically analyzed to obtain the average gradient of important visual markers, the average gradient of redundant visual markers and the average gradient of text markers. Then, they are normalized to obtain the first weight coefficient, the second weight coefficient and the third weight coefficient, respectively. In the post-training quantization process of the multimodal large model, with weighted reconstruction error as the optimization objective, the error related to important visual markers is multiplied by the first weight coefficient, the error related to redundant visual markers is multiplied by the second weight coefficient, and the error related to text markers is multiplied by the third weight coefficient. This is achieved by optimizing the scaling matrix. The value of minimizes the weighted reconstruction error, and the model weights and activations are quantized with low bits. The multimodal large model, after post-training quantization based on the importance of visual markers, is used to perform visual-language task inference.

2. The method as described in claim 1, characterized in that, The step-by-step gradient information calculation includes: Loss function for a given calibration sample Based on this, for the first The first sample The hidden representation vector corresponding to each tag Calculate the gradient norm: ； in, express norm, the gradient norm For marking In the sample Important indicators in the process.

3. The method as described in claim 1, characterized in that, The selection of visual markers whose gradient importance metric ranks in the top 30% as important visual markers includes: Let the first The total number of visual markers in the sample is The visual markers in the sample are categorized as follows: Sort by largest to smallest, and take the first few. A set of visual markers constitutes an important set of visual markers. The remaining visual markers constitute a redundant set of visual markers. ,in, This indicates the rounding up operation.

4. The method as described in claim 3, characterized in that, The step of averaging the gradient importance measures of important visual markers, redundant visual markers, and text markers to obtain the average gradient of important visual markers, the average gradient of redundant visual markers, and the average gradient of text markers includes: The first Key visual markers of the sample Redundant visual markers and average gradient of text tags They are represented as follows: ； ； ； in, For the first A set of text tags in the sample.

5. The method as described in claim 4, characterized in that, The normalization process yields the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient, including: The weighting coefficients for important visual markers, redundant visual markers, and text markers are respectively expressed as: ； ； ； And satisfy: 。 6. The method as described in claim 1, characterized in that, The low-bit quantization of model weights and activations includes: For any input floating-point value The linear uniform quantization operator Represented as: ； Where s is the quantization step size and z is the zero offset. This indicates the rounding operation; and These are the minimum and maximum integer values ​​determined by the quantization bit width; The truncation function is defined as follows: .

7. The method as described in claim 6, characterized in that, The method includes: For signed Bit quantization, selected: 。 8. The method as described in claim 6, characterized in that, The method includes: Given a tensor to be quantized Under the conditions, and Calculate the minimum and maximum values ​​of the tensor to be quantized according to the following preset rules: Then let ； in, This indicates the rounding operation.

9. The method as described in claim 1, characterized in that, The method includes: For a certain linear layer weight tensor In the scaling matrix Under the influence of [the specific function], the weighted reconstruction error objective function of the linear layer is expressed as: in, , and The first The important visual marker input matrix, redundant visual marker input matrix, and text marker input matrix of each sample. express Norm.

10. A multimodal large model quantization system based on visual tag importance weighting, used to implement the method as described in claims 1-9, characterized in that: The system includes: The acquisition module is used to acquire a visual-language sample set for quantization calibration, input the visual-language sample set into a trained multimodal large model, and obtain the output of the multimodal large model and the corresponding loss value. The gradient metric module is used to perform a backpropagation of the loss value relative to the intermediate label representation of the layer to be quantized in the multimodal large model without updating the parameters of the multimodal large model, calculate the gradient information for each label, and obtain the gradient importance metric of each visual label and text label in each sample of the visual-language sample set. The sorting and selection module is used to sort the visual markers according to the gradient importance metric for each sample, and select the visual markers whose gradient importance metric is within the top preset percentage range as important visual markers, and the remaining visual markers as redundant visual markers. The statistics module is used to perform average statistics on the gradient importance measures of important visual markers, redundant visual markers and text markers respectively, to obtain the average gradient of important visual markers, the average gradient of redundant visual markers and the average gradient of text markers, and then normalize them to obtain the first weight coefficient, the second weight coefficient and the third weight coefficient respectively. The bit quantization module is used, during the post-training quantization process of the multimodal large model, to optimize the weighted reconstruction error by multiplying the error related to important visual markers by the first weight coefficient, the error related to redundant visual markers by the second weight coefficient, and the error related to text markers by the third weight coefficient, and by optimizing the scaling matrix. The value of minimizes the weighted reconstruction error, and the model weights and activations are quantized with low bits. The execution inference module is used to perform visual-language task inference using the multimodal large model after post-training quantization processing based on the importance of visual tags.

Images