Vision language model quantization method and system based on nonlinear symmetric mixed precision

Abstract

The application discloses a kind of visual language model quantization method and system based on nonlinear symmetric mixed precision, it is related to artificial intelligence and deep learning model compression optimization technical field, the method includes: obtaining the original weight matrix of VLM;Based on the multimodal architecture characteristics of VLM, the original weight matrix is mapped into the compressed weight matrix of uniform distribution using nonlinear transformation operator;According to the signal flow direction of VLM and multimodal error propagation theory, the difference of network level sensitivity of VLM is identified, and a differentiated precision protection quantization strategy is used for each network layer, that is, the original precision weight of sensitive layer is retained, and the compressed weight matrix of non-sensitive layer is grouped and quantized with symmetric mixed precision;The compressed weight matrix after quantization is converted and packed with unsigned offset and compact bits, and the storage of VLM with high compression ratio is completed.The application realizes high-precision, high-compression-ratio, low-delay quantization compression and storage of VLM, while ensuring the performance of the compressed model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and deep learning model compression and optimization technology, and in particular to a method and system for quantizing visual language models based on nonlinear symmetric mixed precision. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of multimodal artificial intelligence technologies, visual language models, with their core capability of image-text cross-modal semantic alignment, have become a core technology integrating computer vision and natural language processing. Large-scale pre-trained visual language models (VLMs), represented by Qwen2.5-VL, LLaVA, and InternVL, integrate visual encoders and language models to achieve complex tasks such as image understanding, visual question answering, and cross-modal dialogue, showing promising application prospects in fields such as intelligent driving and industrial inspection. The core architecture of VLMs consists of a visual encoder, a word embedding layer, a Transformer cross-modal encoder, and an output prediction layer. The number of parameters typically reaches billions, and storage and computation are performed in high-precision BF16 / FP16 formats. This results in high GPU memory consumption. In long-sequence inference tasks where a single image is encoded into thousands of visual tokens, the computational and storage overhead will further increase. The high deployment cost and inference latency make it difficult to adapt to industrial-grade requirements such as edge devices and real-time interaction. Model compression has become a key bottleneck for its large-scale application.

[0004] Current VLM quantization compression technology faces several core challenges, making it difficult to meet the needs of efficient deployment: On the one hand, multimodal features such as images and text are heterogeneously distributed, and cross-modal alignment is sensitive to errors. Direct linear quantization is prone to precision loss and quantization errors, which can easily lead to model interpretation errors. The additional computation of asymmetric quantization will further amplify the latency of long sequence inference, failing to meet real-time requirements. On the other hand, there is currently a lack of sensitive layer identification and protection strategies adapted to the VLM architecture, and blind quantization can easily lead to a decline in core performance. In addition, uniformly using Int8 or Int4 precision quantization cannot adapt to the differences in precision sensitivity of different levels of the model, making it difficult to balance compression ratio and model performance. Summary of the Invention

[0005] To address the problems of high storage overhead and poor performance in existing VLM quantization compression technologies, such as insufficient multimodal feature adaptation, loss of cross-modal alignment accuracy, low inference efficiency for long sequences, and lack of sensitive layer protection, this invention provides a visual language model quantization method and system based on nonlinear symmetric mixed precision. By performing nonlinear transformation on VLM weights, grouped symmetric mixed precision quantization, sensitive layer protection, and efficient storage and inference, this method achieves high-precision, high-compression-ratio, and low-latency quantization compression and storage of VLM, reducing model storage overhead and inference latency, while ensuring cross-modal alignment accuracy and task performance, thus meeting the high-efficiency computing requirements of terminal deployment and real-time interaction.

[0006] In a first aspect, the present invention provides a method for quantizing visual language models based on nonlinear symmetric mixed precision.

[0007] A visual language model quantization method based on nonlinear symmetric mixed precision includes:

[0008] Obtain the original weight matrix of the large-scale visual language model (VLM) to be quantized and compressed;

[0009] Based on the multimodal architecture characteristics of VLM, a nonlinear transformation operator is used to map the original weight matrix of VLM into a uniformly distributed compressed weight matrix.

[0010] Based on the signal flow and multimodal error propagation theory of VLM, the sensitivity differences of VLM network layers are identified, and a differentiated precision protection quantization strategy is adopted for each network layer: the original precision weights of sensitive layers are retained to maintain model performance; and the compressed weight matrices of non-sensitive layers are subjected to grouped symmetric mixed precision quantization to reduce the space occupied by model storage.

[0011] The quantized compressed weight matrix is ​​subjected to unsigned offset transformation and compact bit packing to complete VLM storage with high compression ratio.

[0012] Further technical solutions also include:

[0013] Based on storage-based quantized compressed VLM weights, hardware-accelerated online inverse quantization inference is performed on the input features, outputting low-latency and efficient inference results, namely:

[0014] The storage-based quantization compression VLM weights employ operator fusion technology, integrating unpacking, offset recovery, scaling, and nonlinear inverse transform into a single CUDA Kernel. After the weights are read from global memory into registers, the weight dequantization process is completed on-chip in one go. At the same time, a shared memory acceleration strategy is adopted to load the quantization parameters in the group quantization into the GPU shared memory.

[0015] The dequantized weight matrix is ​​subjected to mixed-precision matrix operations with the input features, and the long sequence is segmented to output low-latency and efficient inference results.

[0016] A further technical solution utilizes a nonlinear transformation operator to map the original weight matrix of the VLM into a uniformly distributed compressed weight matrix, including:

[0017] Obtain the original FP16 / BF16 precision weight matrix of the large-scale visual language model, separate the weight matrix into a sign matrix and an magnitude matrix, and perform a nonlinear mapping on the magnitude matrix, i.e., apply a square root transformation with a numerical stability constant to the magnitude matrix; the numerical stability constant... The value is 1×10 -9 ;

[0018] The magnitude matrix and sign matrix after nonlinear mapping are recombined to obtain a uniformly distributed compressed weight matrix.

[0019] In a further technical solution, the sensitive layer includes an extremely sensitive layer and a high-sensitivity layer. The extremely sensitive layer includes the network layer of the visual encoder, and the high-sensitivity layer includes the Layer 0,1 cross-modal integration layer and the Layer L-1 output prediction layer of the language model. The insensitive layer includes the 2≤l≤L-2 intermediate layer of the language model. Wherein, L represents the number of network layers of the language model.

[0020] A further technical solution is that the grouped symmetric mixed precision quantization is as follows: the compressed weight matrix is ​​divided into several sub-blocks along the input dimension with a set number of groups, Int8 symmetric quantization is performed on the attention layer of the VLM, and Int4 symmetric quantization is performed on the linear layer of the VLM, and the quantization parameters are calculated independently for each sub-block during quantization.

[0021] Specifically, the Int8 symmetric quantization is performed on the attention layer of the VLM as follows: calculate the maximum absolute value of any grouped weight sub-block, calculate the scale parameter of the Int8 symmetric quantization based on the maximum absolute value; then perform symmetric quantization on the compressed weight matrix based on the scale parameter to scale the weights to the range of [-127, 127].

[0022] Int4 symmetric quantization is performed on the linear layer of VLM to calculate the maximum absolute value of any grouped weight sub-block, and the scale parameter of Int8 symmetric quantization is calculated based on the maximum absolute value; then, the compressed weight matrix is ​​symmetrically quantized based on the scale parameter to scale the weights to the range [-7,7].

[0023] A further technical solution involves performing unsigned offset transformation and compact bit packing on the quantized compressed weight matrix to achieve high compression ratio VLM storage, including:

[0024] The quantized signed Int4 weight matrix in the interval [-7,7] is converted into an unsigned weight matrix in the interval [1,15].

[0025] Based on the bit operation features of CUDA, two consecutive 4-bit unsigned weights in the unsigned weight matrix in the [1,15] interval are encapsulated into an 8-bit byte, and the quantized and compressed VLM is stored after compact bit packing.

[0026] Secondly, the present invention provides a visual language model quantization system based on nonlinear symmetric mixed precision.

[0027] A visual language model quantization system based on nonlinear symmetric mixed precision includes:

[0028] The model weight acquisition module is used to obtain the original weight matrix of the large-scale visual language model (VLM) to be quantized and compressed.

[0029] The nonlinear weight distribution reshaping module is used to map the original weight matrix of VLM into a uniformly distributed compressed weight matrix based on the multimodal architecture characteristics of VLM using nonlinear transformation operators.

[0030] The sensitive layer identification and quantization module is used to identify the sensitivity differences of VLM network layers based on the signal flow direction and multimodal error propagation theory of VLM. It adopts a differentiated precision protection quantization strategy for each network layer, namely: retaining the original precision weights of sensitive layers to maintain model performance; and performing grouped symmetric mixed precision quantization on the compressed weight matrix of non-sensitive layers to reduce the space occupied by model storage.

[0031] The storage module is used to perform unsigned offset transformation and compact bit packing on the quantized compressed weight matrix to complete VLM storage with a high compression ratio.

[0032] Further technical solutions also include:

[0033] The inference implementation module is used for hardware-accelerated online inverse quantization inference of input features based on storage-based quantized compressed VLM weights, outputting low-latency and efficient inference results, including:

[0034] The storage-based quantization compression VLM weights employ operator fusion technology, integrating unpacking, offset recovery, scaling, and nonlinear inverse transform into a single CUDA Kernel. After the weights are read from global memory into registers, the weight dequantization process is completed on-chip in one go. At the same time, a shared memory acceleration strategy is adopted to load the quantization parameters in the group quantization into the GPU shared memory.

[0035] The dequantized weight matrix is ​​subjected to mixed-precision matrix operations with the input features, and the long sequence is processed in blocks to output low-latency and efficient inference results.

[0036] Thirdly, the present invention also provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-described visual language model quantization method based on nonlinear symmetric mixed precision when executing the executable instructions stored in the memory.

[0037] Fourthly, the present invention also provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-described visual language model quantization method based on nonlinear symmetric mixed precision.

[0038] The above one or more technical solutions have the following beneficial effects:

[0039] 1. This invention provides a visual language model quantization method and system based on nonlinear symmetric mixed precision. Through the collaborative design of nonlinear weight distribution reshaping, grouped symmetric mixed precision strategy, sensitive layer topology protection, and hardware-level inference optimization, it achieves a significant reduction in model storage overhead and inference latency, while ensuring cross-modal alignment accuracy and task performance. It meets the high-efficiency computing requirements of terminal deployment and real-time interaction, and solves the problems of multimodal feature heterogeneity, cross-modal alignment sensitivity, and long sequence inference efficiency when quantizing large-scale visual language models (VLMs), thus achieving a balance between model storage, accuracy, and speed.

[0040] 2. To address the challenge of extremely long-tailed distribution of weights in cross-modal projection layers of large-scale visual language models, this invention designs a sign-preserving square root transformation method to achieve precision redistribution in semantic perception. This method compresses large weights through nonlinear transformation, suppressing excessive stretching of the quantization range by outliers and effectively protecting the topological structure of cross-modal attention. Simultaneously, it expands small weights, solving the problem of fine-grained feature loss caused by insufficient resolution near zero in linear quantization. Mathematically, this significantly reduces the total mean square error under long-tailed distributions and greatly improves the cross-modal alignment accuracy after quantization.

[0041] 3. This invention constructs a task-aware hybrid precision quantization strategy based on the Int8 of the Attention layer and the Int4 of the FFN layer. Combined with grouped quantization and Int4 weight compaction technology, it achieves ultimate model compression and storage optimization. While ensuring the core alignment capability of the Attention layer, it fully utilizes the parameter redundancy of the FFN layer to achieve a 4x increase in storage density, reducing the overall memory usage of 7B parameter models by more than 60%. Fine-grained grouping effectively isolates the influence of local outliers, enabling large-scale VLMs to adapt to memory-constrained consumer-grade graphics cards and edge computing devices.

[0042] 4. This invention utilizes the passivation effect of nonlinear transformation on outliers and innovatively employs symmetric quantization to adapt to the highly sparsity characteristics of VLM, decisively improving the efficiency of long sequence inference. By eliminating the zero-point overhead in asymmetric quantization, this method achieves strict "zero-point fidelity," not only simplifying the arithmetic instructions of the dequantization process to a single multiplication, reducing the computational load by approximately 50%, but also preserving the structural zero values ​​of VLM to trigger the physical skip mechanism of the inference engine, significantly improving the throughput of long sequence inference, while reducing metadata transport bandwidth and effectively alleviating the memory wall bottleneck.

[0043] 5. This invention establishes a hierarchical protection mechanism based on topology sensitivity, defining a safe boundary for cross-modal alignment and effectively preventing the cascading amplification of multimodal errors. By preserving full precision in the visual encoder, zero loss of visual perception capability is achieved at a minimal storage cost; simultaneously, enhanced protection is implemented for the first and last key layers of the language model, reducing cross-modal alignment errors and maintaining the output Logits distribution similarity above 98%, effectively avoiding model "illusions" and answer sorting errors.

[0044] 6. This invention designs a full-process hardware operator fusion and shared memory optimization scheme, achieving engineering-level low-latency deployment. By developing a matching CUDA fusion operator, unpacking, dequantization, and nonlinear inverse transform are integrated into a single kernel, and on-chip computation using registers doubles the bandwidth utilization. Combined with shared memory broadcast technology, global memory access for scale parameters is significantly reduced, eliminating memory access latency during dequantization, which can meet the stringent requirements of low-latency inference in real-time interactive scenarios.

[0045] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0046] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0047] Figure 1 This is a flowchart of the visual language model quantization method based on nonlinear symmetric hybrid precision in Embodiment 1 of the present invention;

[0048] Figure 2 This is a flowchart of the nonlinear weight distribution reshaping in Embodiment 1 of the present invention;

[0049] Figure 3 This is a flowchart of the grouped symmetric hybrid precision quantization in Embodiment 1 of the present invention;

[0050] Figure 4This is a flowchart of the VLM sensitive layer identification and protection in Embodiment 1 of the present invention;

[0051] Figure 5 This is a flowchart illustrating the efficient storage and reasoning implementation in Embodiment 1 of the present invention. Detailed Implementation

[0052] It should be noted that the following detailed descriptions are exemplary and are intended only to describe specific embodiments and to provide further explanation of the invention, and are not intended to limit the scope of exemplary embodiments of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0053] The overall concept proposed in this invention is as follows:

[0054] As pointed out in the background section, existing VLM quantization compression methods suffer from problems such as insufficient multimodal feature adaptation, cross-modal alignment accuracy loss, low inference efficiency for long sequences, lack of sensitive layer protection, and limitations of single-precision quantization strategies. These issues make it difficult to meet the requirements for efficient deployment of quantized VLM models, specifically manifested in the following ways:

[0055] 1) VLM needs to process image features from the visual encoder and text features from the language model simultaneously. The weight distributions of the two types of features differ significantly. Visual features typically have a high dynamic range, while cross-modal projection layers exhibit an extreme long-tail distribution. Existing linear quantization methods assume a uniform weight distribution, but this leads to large quantization errors during cross-modal feature fusion, thereby disrupting the fine alignment relationships in the attention mechanism and causing illusions or misunderstandings.

[0056] 2) A single image is usually encoded into hundreds to thousands of visual tokens, which are concatenated with text tokens to form an ultra-long sequence. The additional zero-point storage and subtraction operations required by existing asymmetric quantization methods will significantly amplify the delay in long sequence inference, making it difficult to meet the real-time response requirements.

[0057] 3) The visual encoder, cross-modal feature integration layer, and output prediction layer of VLM are extremely sensitive to quantization errors. Currently, there is a lack of sensitive layer identification and protection strategies based on the characteristics of VLM architecture. Blind quantization can easily lead to a significant drop in core performance.

[0058] 4) In VLM, the attention layer is responsible for cross-modal alignment and has extremely high accuracy requirements. The feedforward network (FFN) layer is mainly based on feature transformation and has a certain degree of fault tolerance. Using the Int8 or Int4 precision quantization strategy uniformly cannot adapt to this difference in inter-layer sensitivity, and it will be difficult to balance the compression ratio and performance of the model.

[0059] To address the aforementioned issues, this invention proposes a nonlinear symmetric mixed-precision quantization method for large-scale visual language models (VLMs), considering aspects such as feature distribution adaptation, inter-layer precision differentiation, sensitive layer protection, and efficient inference compatibility. It designs an integrated scheme encompassing nonlinear transformation, grouped symmetric mixed-precision quantization, sensitive layer protection, and efficient storage and inference. The aim is to significantly reduce model storage overhead and inference latency while ensuring cross-modal alignment accuracy and task performance through the collaborative design of nonlinear transformation and mixed-precision quantization, thus meeting the high-efficiency computing requirements for terminal deployment and real-time interaction.

[0060] The nonlinear symmetric mixed-precision quantization method for large-scale visual language models (VLM) proposed in this invention, such as... Figure 1 As shown, the core idea is as follows: Guided by the multimodal architecture characteristics of VLM, the heterogeneous weights of VLM are first mapped to a uniform distribution through sign-preserving square root transformation, reducing quantization error; then, based on the difference in layer sensitivity, a grouped symmetric mixed-precision quantization strategy is adopted to implement Int8 and Int4 quantization for the Attention layer and FFN layer, respectively; at the same time, a sensitive layer protection mechanism is established to ensure the accuracy of the visual encoder and the key alignment layer; finally, efficient inference is achieved through Int4 weight packing storage and hardware-friendly dequantization design. The method proposed in this invention is applicable to the storage compression and inference acceleration optimization of visual language models and large language models (LLM) based on the Transformer architecture, and can be widely applied to real-world scenarios that rely on multimodal interaction, including but not limited to intelligent terminal visual question answering, real-time image semantic understanding for autonomous driving, visual document analysis for industrial quality inspection, and multimodal dialogue systems for edge devices.

[0061] Example 1

[0062] This embodiment provides a visual language model quantization method based on nonlinear symmetric mixed precision, specifically including the following steps:

[0063] Step S1: Obtain the original weight matrix of the large-scale visual language model (VLM) to be quantized and compressed;

[0064] Step S2: Based on the multimodal architecture characteristics of VLM, the original weight matrix of VLM is mapped to a uniformly distributed compressed weight matrix using a nonlinear transformation operator.

[0065] Step S3: Based on the signal flow direction and multimodal error propagation theory of VLM, identify the sensitivity differences of VLM network layers, and adopt a differentiated precision protection quantization strategy for each network layer: retain the original precision weights of sensitive layers to maintain model performance; perform grouped symmetric mixed precision quantization on the compressed weight matrix of non-sensitive layers to reduce the space occupied by model storage.

[0066] Step S4: Perform unsigned offset transformation and compact bit packing on the quantized compressed weight matrix to complete the VLM storage with high compression ratio.

[0067] The following content provides a more detailed introduction to the visual language model quantization method based on nonlinear symmetric mixed precision proposed in this embodiment.

[0068] In steps S1 and S2, the original FP16 / BF16 precision weight matrix of the large-scale visual language model (VLM) is obtained, and the weight matrix is ​​reshaped using a nonlinear weight distribution. This nonlinear weight distribution reshaping process is a core pre-process for achieving high-precision quantization. It aims to address the insufficient quantization resolution caused by the extremely long-tailed distribution of weights in the VLM's cross-modal projection layer. By introducing a nonlinear transformation operator, the weight density function is reshaped to adapt to the subsequent uniform quantizer.

[0069] Specifically, a nonlinear transformation operator is used to map the heterogeneously distributed weight matrix of the VLM to a uniformly distributed compressed weight matrix, thus completing the reshaping of the nonlinear weight distribution. The entire process is designed as a closed-loop control process, such as... Figure 2 As shown, the process includes the following steps: First, input analysis is performed, receiving the original FP16 / BF16 weight matrix. Secondly, the sign of the weight matrix is ​​separated, and the sign matrix of the weights is extracted. With amplitude matrix Then, a nonlinear mapping is performed on the magnitude matrix, that is, a square root transformation operator is applied to the magnitude matrix to compress the high dynamic range, and numerical stabilization is performed during the transformation process, that is, a numerical stability constant is introduced. Factors are used to prevent gradient explosion or numerical underflow; finally, the output is a recombined weight matrix, which is obtained by recombining the magnitude matrix and sign matrix after nonlinear mapping to obtain a uniformly distributed compressed weight matrix after transformation. Then it is passed to the quantizer.

[0070] Furthermore, the mathematical modeling process for the above nonlinear transformation is as follows:

[0071] (1) In the forward transformation stage, the weights of the VLM cross-modal projection layer are adjusted. The exhibited peak-and-fat-tailed characteristics necessitate defining a sign-preserving square-root transform function. for:

[0072] (3.1)

[0073] in, Represents element-wise multiplication. Let be the numerical stability constant, and let its value be . Used to avoid in To address the issue of infinite derivatives, we need to ensure gradient stability during backpropagation.

[0074] (2) Inverse recovery: In the dequantization stage of reasoning, through inverse transformation Restoring the original weights to their original numerical scale can be expressed as:

[0075] (3.2)

[0076] Preferably, in the actual implementation of the CUDA Kernel (CUDA is a computing platform, and the Kernel is the data parallel processing function on that platform), to accelerate computation, the following is usually ignored: The minor impact on reasoning accuracy can be simplified as follows:

[0077] (3.3)

[0078] Furthermore, the advantages of the aforementioned nonlinear transformation are mathematically proven as follows:

[0079] The design proposition is: for weights that follow a long-tail distribution... For example, the total mean square error (MSE) generated by quantization using a Laplace distribution or Gaussian distribution is smaller than that generated by linear uniform quantization after the square root nonlinear transformation.

[0080] The proof is as follows: First, define the quantization error model, assuming the total number of quantizer levels is... Quantization interval is The step size of linear quantization is fixed at 1. Nonlinear quantization can be viewed as the transformation of variables. Perform uniform quantization, which is performed in the original domain. Equivalent step size It is dynamic and changing.

[0081] Next, we derive the equivalent step size of the nonlinear transformation. Let the transformation function... Here, we only consider the positive semi-axis; the negative semi-axis is symmetrical. Uniform quantization is performed on the domain with a step size of . According to differential relations The equivalent step size of the original domain Approximately:

[0082] (3.4)

[0083] Therefore, the equivalent step size after nonlinear transformation can be seen. With weight magnitude Proportional, that is: when When it is very small, close to 0, the equivalent step size Extremely small, with extremely high quantization resolution; when When the long tail is large, the equivalent step size is... As the size increases, the quantization resolution decreases.

[0084] Then, the error expectation formula is constructed to quantify the mean square expectation of the error. It can be represented as a probability density function Integrating the local quantization noise power, assuming the quantization noise is uniformly distributed within each interval, the local noise power is proportional to the square of the step size, and can be expressed as:

[0085] (3.5)

[0086] Finally, we compare linear and nonlinear errors. For linear quantization error... ,because const indicates a constant, and VLM weights Concentrating around 0, linear quantization uses an excessively large fixed step size near 0, leading to significant errors in frequently occurring values. The error formula is expressed as:

[0087] (3.6)

[0088] In the above formula, It is the probability density function of the weights. It is the maximum absolute value of the weights and determines the fixed step size of linear quantization. This is the quantization level. The error is completely multiplied by the global maximum. Dominant. Furthermore, as long as there is an outlier far from the center in the distribution, the quantization error benchmark of the entire model will be raised.

[0089] For nonlinear quantization error Substitute We can obtain:

[0090] (3.7)

[0091] As can be seen above, the error is proportional to the expected absolute value of the weights. In a long-tailed distribution, the expected value is... Much smaller than the maximum value .

[0092] Therefore, for the typical long-tailed distribution of VLM, the probability density Follow The error of linear quantization is mainly limited by the fixed step size resolution near 0, while the square root transform greatly reduces the step size near 0. At that time, equivalent step size Quantization precision is concentrated and allocated to probability density. The highest region is found, which statistically significantly reduces the total mean square error, thus completing the verification.

[0093] Based on the aforementioned mathematical properties, this embodiment implements precision redistribution for semantic awareness within the VLM architecture, i.e., lossless preservation of fine-grained features. Specifically, in the fusion layer of the VLM's visual encoder and language model, there are a large number of parameters with small amplitudes that are crucial for semantic fine-tuning. The limitations of linear quantization cause these parameters to typically be smaller than the minimum step size of linear quantization. This results in forced quantization to 0, causing feature loss, while this embodiment can... The expansion effect stretches small weights to the effective dynamic range of the quantizer, preserving fine-grained cross-modal characteristics.

[0094] Meanwhile, in the cross-modal attention matrix, a small number of large weights (outliers) determine the alignment direction of visual tokens and text tokens. This embodiment compresses the large weights through square root transformation. This suppresses the excessive stretching of the quantization range by outliers, preventing the overall accuracy from decreasing due to the expansion of the quantization range by a few outliers, thus protecting the key mapping topology of "image-text".

[0095] In step S3, a grouped symmetric mixed-precision quantization strategy is used to accelerate model compression and inference. Simultaneously, a VLM sensitive layer identification and protection mechanism is introduced to ensure the stability of the topology. This step S3 forms a tight logical loop with step S2: nonlinear transformation solves the microscopic distribution problem of weights, giving the data a physical basis for quantization; mixed-precision quantization solves the macroscopic compression problem of the model by aggressively using Int4 in the intermediate layer to achieve extreme parameter efficiency; VLM sensitive layer identification and protection solves the boundary problem of error propagation. Because VLM has a significant causal chain structure (perception -> alignment -> inference -> decision), if a general quantization strategy is applied indiscriminately to the entire network, small disturbances at the source or end will be amplified layer by layer by the deep network. By identifying and locking down key topology nodes with veto power, a safety boundary is defined for the aggressive compression strategy based on mixed-precision quantization.

[0096] Specifically, before implementing the quantization strategy, based on the differences in layer sensitivity of the VLM, a differentiated precision protection quantization strategy is adopted for each network layer. This preserves the original precision of sensitive layers, while non-sensitive layers undergo the aforementioned grouped symmetrical mixed precision quantization. For example... Figure 4 As shown, the VLM sensitive layer identification and protection process is as follows:

[0097] First, the rules for identifying sensitive layers are defined. Based on the signal flow and multimodal error propagation theory of VLM, the VLM network layers are divided into "error sources," "error amplifiers," and "error tolerance zones," and the following sensitive layers are identified accordingly:

[0098] (1) The visual encoder is an irreversible source of perception. The visual encoder (ViT) is responsible for processing... The original pixel space is mapped to high-dimensional semantic features. This layer is sensitive because it is the starting point of information in the entire chain. According to the data processing inequality in information theory, subsequent language models (LLM) cannot recover the image details lost in the encoding stage. Introducing quantization noise here is equivalent to putting "colored glasses" on all subsequent inference processes. This feature loss is irreversible in the system, therefore all Vision Transformer layers are extremely sensitive layers.

[0099] (2) The language model consists of network layers at the beginning and end that implement cross-modal alignment and decision-making. Layer 0 and 1 are cross-modal integration layers (i.e., cross-modal encoders) and contain visual features. Text features At the intersection of visual and textual representations, if a quantization error occurs, it will cause the projection of the visual token in the language manifold space to drift, directly disrupting the semantic alignment between image and text. This is the root cause of the illusion generated by VLM. Furthermore, Layer L-1 is the output prediction layer and the final output of the Logits distribution. In visual tasks, the probability of the top-1 word is often only slightly different from other candidate words. A small perturbation in the weights of the final layer is enough to change the relative ranking of Logits, causing the output to change from "yes" to "no". Therefore, Layers 0, 1 and L-1 are highly sensitive layers, while the remaining layers are insensitive layers.

[0100] Next, based on the sensitivity of the identified layers, a differentiated precision protection quantization strategy is adopted for each network layer. Specifically, to achieve Pareto optimality between maintaining accuracy and compressing the model, this embodiment proposes a hierarchical protection strategy: the visual encoder and language model sensitive layers adopt a full-precision immune strategy, which means completely preserving the original BF16 / FP16 precision without any form of quantization compression. The rationale for this strategy is that although the visual encoder accounts for about 10%-15% of the parameters, the marginal benefit of compressing it is far less than its marginal cost. Preserving full precision ensures the purity of the input information, allowing the subsequent quantization of the language model to be based on high-quality features. This language model sensitive layer processing achieves cross-modal alignment and numerical stability of the final decision with minimal storage cost. The non-sensitive intermediate layers, on the other hand, adopt an aggressive mixed-precision strategy, i.e., the intermediate layers of the language model (… The Attention Int8+FFN Int4 quantization strategy is implemented. The basis of this strategy is that the error of the middle layer is limited to a controllable range due to the high-precision clamping at both ends, so that it can safely undertake the main model compression task.

[0101] Furthermore, the rationality of the above protection strategy is verified. This embodiment demonstrates the scientific validity of the strategy from the perspective of parameter-performance ratio and system robustness.

[0102] The parameter-efficiency gain is non-linear, and experiments show that the performance of VLM is non-linearly sensitive to parameter accuracy. Forcing Int8 quantization on the visual encoder, while saving less than 0.7GB of GPU memory, leads to a precipitous drop in visual task accuracy. This embodiment sacrifices this 5% compression space in exchange for zero loss of visual perception capability. This is a cost-effective strategic sacrifice that ensures the overall usability of the compression scheme.

[0103] Meanwhile, the aforementioned strategy can suppress alignment collapse. Comparative experiments revealed that without enhancement protection of Layer 0,1, the heatmaps of the attention matrices for visual and text tokens exhibit significant diffusion, meaning the model cannot focus on key regions in the image. However, after applying this protection strategy, the attention focus is highly consistent with the full-precision model. This demonstrates that the strategy can effectively truncate the cascading amplification of quantization noise at the cross-modal interface, maintaining the topological stability of the image-text mapping.

[0104] Furthermore, the above strategy ensures the fidelity of the decision boundary. For the output layer (i.e., Layer L-1), the protection strategy essentially maintains the confidence ranking obtained by the model after the end-to-end inference. Validation data shows that this strategy significantly reduces the KL divergence between the Logits distribution of the quantized model output and the original model, effectively avoiding answer flipping errors caused by insufficient numerical precision.

[0105] As a further implementation, the compressed weight matrix of the aforementioned non-sensitive layers is subjected to grouped symmetric mixed-precision quantization to reduce the space occupied by the model storage. In this embodiment, the design of the grouped symmetric mixed-precision quantization strategy is as follows: based on the functional differences and parameter sensitivity analysis of different layers of the VLM, this embodiment adopts a task-aware mixed-precision strategy of "Attention layer Int8 + FFN layer Int4", while combining grouped quantization technology to adapt to multi-scale features, and rigorously demonstrating the optimal adaptability of symmetric quantization under this architecture from the perspective of mathematical distribution and computational complexity. Figure 3 As shown, the precision quantization process for this grouped symmetric mixture is as follows:

[0106] First, a quantization grouping strategy is implemented. To address the significant differences in dynamic range among different feature channels in VLM, the compressed weight matrix after nonlinear transformation is... Implement fine-grained grouping along the input dimension, defined as follows: Let the transformed weight matrix be... ,in Number of output channels Set the group size to the number of input channels. As the default value, along the input dimension Divide the matrix into A small block, This indicates rounding up. Specifically, for the ... Groups, their weight subsets It has its own set of quantitative parameters.

[0107] The quantization grouping strategy described above can effectively isolate the influence range of outliers. If an extreme large value exists in a certain channel, it will only stretch the quantization scale within that group. This avoids "polluting" the quantization resolution of the entire layer. This is especially crucial for hybrid inputs in VLM that combine high dynamic range visual features with discrete text embeddings.

[0108] Then, mixed precision quantization is performed. This embodiment designs a non-uniform precision allocation scheme based on the precision sensitivity gradient of the internal components of the VLM, including performing Int8 symmetric quantization on the attention layer of the VLM and performing Int4 symmetric quantization on the linear layer of the VLM.

[0109] (1) Attention Layer Int8 Quantization: The core function of cross-modal alignment is implemented based on the Attention layer of VLM, which is responsible for calculating the visual token. ) and Language Token The correlation score matrix of ) This process is the "nerve center" of cross-modal alignment and is extremely sensitive to numerical perturbations. For the Attention layer, Int8 (i.e., 8-bit) quantization is used, providing 256 discrete energy levels and retaining sufficient precision margin to maintain the alignment topology. Specifically, during quantization, for any grouping weights... Calculate its maximum absolute value The scale parameter of Int8 symmetric quantization is calculated based on the maximum absolute value, and is as follows:

[0110] (3.8)

[0111] Then, symmetric quantization is performed on the compressed weight matrix according to the scale parameter, scaling the weights to the [-127, 127] interval. The quantization formula is as follows:

[0112] (3.9)

[0113] Next, quantize the results of all groups. By concatenating the components, the final quantized weights are obtained. This section uses signed symmetric ranges. Avoid using This ensures complete symmetry between the positive and negative intervals.

[0114] (2) Int4 Quantization of the FFN Layer: The nonlinear feature transformation is implemented based on the FFN layer (i.e., the linear layer) of the VLM, which includes Gate / Up / Down projection, expanding the dimension to four times that of the hidden layer before compression. Since the input is the fused feature and the parameters have significant channel redundancy, it has strong fault tolerance. Therefore, Int4 (4-bit) quantization is used for this FFN layer, using only 15 discrete energy levels to achieve the maximum compression ratio. Similarly, during the quantization process, arbitrary grouping weights... Calculate its maximum absolute value The scale parameter of Int8 symmetric quantization is calculated based on the maximum absolute value, and is as follows:

[0115] (3.10)

[0116] The compressed weight matrix is ​​symmetrically quantized according to the scale parameter, scaling the weights to the [-7,7] interval. The quantization interval is:

[0117] (3.11)

[0118] Next, quantize the results of all groups. By concatenating the components, the final quantized weights are obtained. .

[0119] Traditional 4-bit quantization typically employs asymmetric quantization to fully utilize all discrete scales. However, the 4-bit quantization in this embodiment uses symmetric quantization, which leverages the compatibility between "sign-preserving square root transform" and "symmetric quantization," as well as the compatibility between "the high sparsity exhibited by the VLM model" and "symmetric quantization."

[0120] Specifically, traditional symmetric quantization has a weakness: symmetric quantization uses a symmetric interval centered at 0. To cover the data, its quantization step size is entirely determined by the absolute maximum value in the data. The decision is:

[0121] (3.12)

[0122] Where B represents the number of bits quantized.

[0123] In the cross-modal projection layer of VLM, the weights exhibit an extreme long-tailed distribution, with a very small number of outliers with extremely large absolute values. If symmetric quantization is used directly, it would be difficult to cover this large outlier. This forces the quantization interval to be stretched extremely large. However, the vast majority of weights are actually concentrated in a very small range near 0, resulting in most of the quantizer's precision (i.e., quantization grid points) being wasted on covering 0 to 0. In the vacuum zone between these areas, the quantization resolution of the effective weights is extremely low, resulting in a waste of precision.

[0124] This embodiment utilizes the adaptation of nonlinear transformation and symmetric quantization: the nonlinear transformation has a passivation effect on outliers, and this embodiment introduces the transformation through the nonlinear weight distribution reshaping in step S2. It has a significant compression effect on large values ​​(derivative < 1) and an expansion effect on small values ​​(derivative > 1). This method achieves two main goals: First, it reduces the degree of outliers. Assuming the original outlier is 100, the normal value is 0.01, and the ratio is 10000:1, after transformation, the outlier becomes 10, the normal value becomes 0.1, and the ratio becomes 100:1. Second, it maximizes the utilization of accuracy. After transformation... It is significantly pulled closer to the central region. Based on this, symmetrical quantization is then applied, which determines the quantization range. The quantization grid, which was previously bloated by outliers, is now densely distributed within the effective data region. Therefore, the nonlinear transformation eliminates the excessive sensitivity of symmetric quantization to outliers, thus removing the disadvantage of symmetric quantization's waste of half-side precision. The combination of these two approaches retains the computational efficiency of symmetric quantization while achieving the distribution adaptability of near-nonlinear quantization.

[0125] Besides distribution adaptation, this embodiment demonstrates that symmetric quantization is the most feasible mathematical choice to address the high sparsity exhibited by VLM models in actual inference. Specifically, VLM has three types of zero-value characteristics, meaning that large-scale visual language models generate a large number of zero values ​​with clear physical meaning during inference: First, structural zeros (padding zeros): When VLM processes multi-graph, multi-turn dialogues, it needs to pad indefinite-length sequences, resulting in a large number of padding zeros in the input tensor; second, activation zeros: The ReLU or ReLU-like activation functions widely used in FFN layers truncate negative values ​​to strict zeros; and third, mask zeros: The attention mask mechanism causes a large number of attention weights to be set to 0 to mask irrelevant token associations.

[0126] Secondly, the "zero-point fidelity" effect differs between symmetric quantization and asymmetric quantization.

[0127] The formula for asymmetric quantization mapping is:

[0128] (3.13)

[0129] in, This represents the result after mapping. To represent the weights before mapping, This represents the scale parameter, where Z is the bias.

[0130] because It is based on and The result is obtained by rounding down after floating-point calculations, which introduces zero-point quantization error. As a result, a perfect "0" in the input becomes tiny non-zero noise, such as 0.002, after the quantization-dequantization loop. This can lead to invalid calculations in the padding area and even interfere with the masking mechanism of Attention, triggering an avalanche effect.

[0131] The formula for symmetric quantization mapping is:

[0132] (3.14)

[0133] No additive bias is introduced in the above process. Its advantage is that the real number 0.0 is always mapped to the integer 0, and it remains strictly 0.0 after dequantization. Therefore, only symmetric quantization can perfectly preserve the sparse structure of VLM.

[0134] Finally, the decisive impact of symmetric quantization on VLM inference efficiency is analyzed. The symmetric quantization strategy adopted in this invention ( This not only adapts to VLM in terms of mathematical distribution, but also eliminates asymmetric quantization formulas. Zero point item in This resulted in a decisive efficiency improvement in three dimensions: computational complexity, storage overhead, and sparsity acceleration. The significant reduction in dequantization computational complexity, particularly in the core matrix multiplication of inference (… Before asymmetric quantization, a bias correction term needs to be introduced. ), that is, calculation In contrast, the symmetric quantization in this embodiment only requires performing... This will achieve instruction-level acceleration, eliminating the "subtract zero" operation required for asymmetric quantization. In the GPU core's dequantization kernel, single-weight arithmetic instructions are simplified from "subtraction + multiplication" to a single "multiplication," reducing dequantization computation by approximately 50% and significantly improving the efficiency of the memory read and computation pipeline during the decoding stage.

[0135] Furthermore, the above method also optimizes storage overhead and bandwidth. Based on a 7B parameter-scale VLM and a group size of 128, removing the independent FP16 zero-point parameters for each group directly reduces parameter storage requirements by approximately 109MB. This results in a bandwidth advantage; during memory-intensive inference processes, this translates to a reduction of 109MB in metadata movement, effectively alleviating the memory wall bottleneck in long-sequence inference with VLMs.

[0136] The zero-point fidelity of the aforementioned methods is crucial to the inference performance of VLM. Regarding sparse computation acceleration, modern inference engines such as vLLM and TensorRT-LLM utilize sparsity for computational pruning. If 0 becomes 0.002, the GPU must perform a complete matrix multiplication, unable to skip invalid blocks, severely slowing down long sequence inference. Under symmetric quantization, zero-value compression encoding is extremely efficient, maintaining the purity of zero values ​​and helping to further reduce GPU memory usage using sparse storage formats.

[0137] In step S4, in order to minimize the memory usage, this embodiment designs a storage format of "unsigned offset + compact bit packing". The quantized compressed weight matrix is ​​converted by unsigned offset and stored in compact bit packing. The Int4 weights are adapted to the standard byte storage system to obtain the VLM model weights after mixed precision quantization and store them.

[0138] Specifically, in the Int4 weight packing and storage mechanism, the unsigned offset is converted into: the quantized signed Int4 weight. Since it cannot be directly stored using standard data types, we first apply a positive offset to it and convert it to an unsigned integer, as follows:

[0139] (3.15)

[0140] After conversion It avoids the value of 0, retains 0 for sparse representation or padding detection, and is compatible with the range of 4-bit unsigned representation.

[0141] Next, bit-packing is performed. This utilizes CUDA's bit manipulation capabilities to encapsulate every two consecutive unsigned 4-bit values ​​into a standard 8-bit byte (uint8), achieving a 2:1 physical compression. During compression, low-order bits are padded with the weights of even-numbered indices in the sequence. Stored in the lower 4 bits (0-3 bits) of the byte; higher bits padding: weight of odd indices in the sequence. Stored in the high 4 bits (4-7 bits) of the byte, the packing formula is:

[0142] (3.16)

[0143] In terms of storage benefits, the above method can increase the storage density of weights to four times that of the BF16 format. Taking the 7B model as an example, the memory usage of the FFN layer drops sharply from 5.6GB to 1.4GB, making it suitable for the limited memory environments of consumer-grade graphics cards and edge devices.

[0144] As a further implementation, based on the stored quantized compressed VLM weights, hardware-accelerated online dequantization inference is performed on the input features to output low-latency and efficient inference results.

[0145] Specifically, in conjunction with step S4 above, this embodiment designs a full-link optimization scheme from physical storage to computing core to address the storage bottleneck of large-scale parameters in VLM and the computational latency challenge of long sequence inference. This scheme integrates compact packing storage technology with Int4 weights and fusion dequantization operator based on CUDA hardware acceleration. While achieving 4 times storage compression, it significantly improves inference throughput through shared memory optimization and operator fusion.

[0146] like Figure 5 As shown, inverse quantization inference is performed based on the stored quantized VLM model weights and input features, i.e., online computation. Considering that the traditional inverse quantization process involves multiple independent steps—"unpacking-offset recovery-scaling-nonlinear transformation"—frequent GPU memory reads and writes are the main source of inference latency, this embodiment uses CUDA code to perform hardware-accelerated fused inverse quantization inference, achieving full-process operator fusion and GPU memory access optimization, including:

[0147] (1) Using operator fusion (Kernel Fusion) technology, unpacking, offset recovery, scaling and nonlinear inverse transformation are integrated into a single CUDA Kernel. After the weights are read from the global video memory into the register, all transformations are completed on-chip at once, that is, the weight inverse quantization is completed, and the BF16 / FP16 data for matrix operations is directly output.

[0148] The above method enables single memory access, meaning the kernel reads uint8 packaged data once and can process the unpacking and calculation of two weight values ​​in parallel in the register, improving bandwidth utilization by 100%. It also implements fused computing logic, i.e., through... and Quick Extraction Implement bitwise operation unpacking; execute in registers Symmetric offset recovery is achieved by directly restoring to the linear distribution space; execution is performed immediately in the register. The nonlinear inverse transformation restores the original weight order of magnitude distribution.

[0149] (2) To address the characteristic of scale parameters being repeatedly read by multiple weights within a group during grouped quantization, an explicit shared memory acceleration strategy is employed to achieve cooperative loading. Specifically, at the initial startup of the CUDA Block, threads within the same block collaboratively load all scale parameters for the current row into the high-speed GPU shared memory at once. This method uses an on-chip broadcast mechanism; in subsequent calculations, threads directly read the scale from the low-latency shared memory, avoiding repeated access to global GPU memory. For a group_size=128 configuration, this strategy reduces the global GPU memory access volume of scale parameters by two orders of magnitude, eliminating the bandwidth bottleneck during dequantization.

[0150] (3) The weight matrix after dequantization is subjected to mixed precision matrix operation with the input features, and the long sequence is processed in blocks to ensure the accuracy and efficiency of long sequence reasoning, and finally output low-latency and efficient reasoning results.

[0151] Specifically, this method achieves dynamic precision alignment, i.e., input features Weights generated by real-time dequantization Performing a standard linear transformation in Tensor Core is as follows:

[0152] (3.17)

[0153] in, This indicates the bias. Since the output of the dequantization kernel is directly aligned with the precision type of the input features, it avoids the overhead of additional type conversion before computation.

[0154] Furthermore, for scenarios where VLM processes long sequences of thousands of tokens, the system employs block matrix multiplication to achieve block-based processing of long sequences. Combined with the aforementioned fusion kernel, weights are dynamically decompressed into L2 cache or registers in blocks (tiles), and released immediately after calculation. Peak memory usage no longer increases linearly with sequence length, effectively preventing memory overflow.

[0155] Example 2

[0156] This embodiment provides a visual language model quantization system based on nonlinear symmetric mixed precision, including:

[0157] The model weight acquisition module is used to obtain the original weight matrix of the large-scale visual language model (VLM) to be quantized and compressed.

[0158] The nonlinear weight distribution reshaping module is used to map the original weight matrix of VLM into a uniformly distributed compressed weight matrix based on the multimodal architecture characteristics of VLM using nonlinear transformation operators.

[0159] The sensitive layer identification and quantization module is used to identify the sensitivity differences of VLM network layers based on the signal flow direction and multimodal error propagation theory of VLM. It adopts a differentiated precision protection quantization strategy for each network layer, namely: retaining the original precision weights of sensitive layers to maintain model performance; and performing grouped symmetric mixed precision quantization on the compressed weight matrix of non-sensitive layers to reduce the space occupied by model storage.

[0160] The storage module is used to perform unsigned offset transformation and compact bit packing on the quantized compressed weight matrix to complete VLM storage with a high compression ratio.

[0161] Furthermore, the system proposed in this embodiment also includes:

[0162] The inference implementation module is used for storage-based quantization compression of VLM weights, hardware-accelerated online dequantization inference of input features, and outputting low-latency and efficient inference results.

[0163] Specifically, in this inference implementation module, the storage-based quantized compressed VLM weights employ operator fusion technology to integrate unpacking, offset recovery, scaling, and nonlinear inverse transformation into a single CUDA Kernel. After the weights are read from global video memory into registers, the weight dequantization process is completed on-chip in one go. At the same time, a shared memory acceleration strategy is adopted to load the quantization parameters in the group quantization into the GPU shared memory. The dequantized weight matrix performs mixed-precision matrix operations with the input features, and long sequences are processed in blocks to output low-latency and efficient inference results.

[0164] Example 3

[0165] This embodiment provides an electronic device, including: a memory for storing executable instructions; and a processor for executing the executable instructions stored in the memory to implement the method provided in this embodiment.

[0166] Example 4

[0167] This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, will cause the processor to execute the method described above in this embodiment.

[0168] The steps and methods involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0169] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0170] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention has been described in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the present invention.

Claims

1. A method for quantizing visual language models based on nonlinear symmetric hybrid precision, characterized in that, include: Obtain the original weight matrix of the large-scale visual language model (VLM) to be quantized and compressed; Based on the multimodal architecture characteristics of VLM, a nonlinear transformation operator is used to map the original weight matrix of VLM into a uniformly distributed compressed weight matrix. Based on the signal flow and multimodal error propagation theory of VLM, the sensitivity differences of VLM network layers are identified, and a differentiated precision protection quantization strategy is adopted for each network layer: the original precision weights of sensitive layers are retained to maintain model performance; and the compressed weight matrices of non-sensitive layers are subjected to grouped symmetric mixed precision quantization to reduce the space occupied by model storage. The quantized compressed weight matrix is ​​subjected to unsigned offset transformation and compact bit packing to complete VLM storage with high compression ratio; Based on storage-based quantized compressed VLM weights, hardware-accelerated online inverse quantization inference is performed on the input features, outputting low-latency and efficient inference results, namely: The storage-based quantization compression VLM weights employ operator fusion technology, integrating unpacking, offset recovery, scaling, and nonlinear inverse transform into a single CUDA Kernel. After the weights are read from global memory into registers, the weight dequantization process is completed on-chip in one go. At the same time, a shared memory acceleration strategy is adopted to load the quantization parameters in the group quantization into the GPU shared memory. The dequantized weight matrix is ​​subjected to mixed-precision matrix operations with the input features, and the long sequence is processed in blocks to output low-latency and efficient inference results. The grouped symmetric mixed precision quantization is as follows: the compressed weight matrix is ​​divided into several sub-blocks along the input dimension with a set number of groups, Int8 symmetric quantization is performed on the attention layer of VLM, and Int4 symmetric quantization is performed on the linear layer of VLM, and the quantization parameters are calculated independently for each sub-block during quantization. Specifically, the Int8 symmetric quantization is performed on the attention layer of the VLM as follows: calculate the maximum absolute value of any grouped weight sub-block, calculate the scale parameter of the Int8 symmetric quantization based on the maximum absolute value; then perform symmetric quantization on the compressed weight matrix based on the scale parameter to scale the weights to the range of [-127, 127]. Int4 symmetric quantization is performed on the linear layer of VLM to calculate the maximum absolute value of any grouped weight sub-block, and the scale parameter of Int8 symmetric quantization is calculated based on the maximum absolute value; then, the compressed weight matrix is ​​symmetrically quantized based on the scale parameter to scale the weights to the range [-7,7].

2. The visual language model quantization method based on nonlinear symmetric hybrid precision as described in claim 1, characterized in that, Using a nonlinear transformation operator, the original weight matrix of the VLM is mapped to a uniformly distributed compressed weight matrix, including: Obtain the original FP16 / BF16 precision weight matrix of the large-scale visual language model, separate the weight matrix into a sign matrix and an magnitude matrix, and perform a nonlinear mapping on the magnitude matrix, i.e., apply a square root transformation with a numerical stability constant to the magnitude matrix; the numerical stability constant... The value is 1×10 -9 ; The magnitude matrix and sign matrix after nonlinear mapping are recombined to obtain a uniformly distributed compressed weight matrix.

3. The visual language model quantization method based on nonlinear symmetric hybrid precision as described in claim 1, characterized in that, The sensitive layer includes an extremely sensitive layer and a highly sensitive layer. The extremely sensitive layer includes the network layer of the visual encoder, and the highly sensitive layer includes the Layer 0,1 cross-modal integration layer and the Layer L-1 output prediction layer of the language model. The insensitive layer includes the 2≤1≤L-2 intermediate layer of the language model.

4. The visual language model quantization method based on nonlinear symmetric hybrid precision as described in claim 1, characterized in that, The quantized compressed weight matrix undergoes unsigned offset transformation and compact bit packing to achieve high-compression VLM storage, including: The quantized signed Int4 weight matrix in the interval [-7,7] is converted into an unsigned weight matrix in the interval [1,15]. Based on the bit operation features of CUDA, two consecutive 4-bit unsigned weights in the unsigned weight matrix in the [1,15] interval are encapsulated into an 8-bit byte, and the quantized and compressed VLM is stored after compact bit packing.

5. A visual language model quantization system based on nonlinear symmetric hybrid precision, characterized in that, include: The model weight acquisition module is used to obtain the original weight matrix of the large-scale visual language model (VLM) to be quantized and compressed. The nonlinear weight distribution reshaping module is used to map the original weight matrix of VLM into a uniformly distributed compressed weight matrix based on the multimodal architecture characteristics of VLM using nonlinear transformation operators. The sensitive layer identification and quantization module is used to identify the sensitivity differences of VLM network layers based on the signal flow direction and multimodal error propagation theory of VLM. It adopts a differentiated precision protection quantization strategy for each network layer, namely: retaining the original precision weights of sensitive layers to maintain model performance; and performing grouped symmetric mixed precision quantization on the compressed weight matrix of non-sensitive layers to reduce the space occupied by model storage. The storage module is used to perform unsigned offset transformation and compact bit packing on the quantized compressed weight matrix to complete VLM storage with a high compression ratio. The inference implementation module is used for hardware-accelerated online inverse quantization inference of input features based on storage-based quantized compressed VLM weights, outputting low-latency and efficient inference results, including: The storage-based quantization compression VLM weights employ operator fusion technology, integrating unpacking, offset recovery, scaling, and nonlinear inverse transform into a single CUDA Kernel. After the weights are read from global memory into registers, the weight dequantization process is completed on-chip in one go. At the same time, a shared memory acceleration strategy is adopted to load the quantization parameters in the group quantization into the GPU shared memory. The dequantized weight matrix is ​​subjected to mixed-precision matrix operations with the input features, and the long sequence is processed in blocks to output low-latency and efficient inference results. The grouped symmetric mixed precision quantization is as follows: the compressed weight matrix is ​​divided into several sub-blocks along the input dimension with a set number of groups, Int8 symmetric quantization is performed on the attention layer of VLM, and Int4 symmetric quantization is performed on the linear layer of VLM, and the quantization parameters are calculated independently for each sub-block during quantization. Specifically, the Int8 symmetric quantization is performed on the attention layer of the VLM as follows: calculate the maximum absolute value of any grouped weight sub-block, calculate the scale parameter of the Int8 symmetric quantization based on the maximum absolute value; then perform symmetric quantization on the compressed weight matrix based on the scale parameter to scale the weights to the range of [-127, 127]. Int4 symmetric quantization is performed on the linear layer of VLM to calculate the maximum absolute value of any grouped weight sub-block, and the scale parameter of Int8 symmetric quantization is calculated based on the maximum absolute value; then, the compressed weight matrix is ​​symmetrically quantized based on the scale parameter to scale the weights to the range [-7,7].

6. An electronic device, characterized in that, include: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the visual language model quantization method based on nonlinear symmetric mixed precision as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The device stores executable instructions for causing a processor to execute the executable instructions to implement the visual language model quantization method based on nonlinear symmetric mixed precision as described in any one of claims 1-4.

Images