Quantization methods, devices, equipment, media, and programs for visual language models

Abstract

This invention discloses a method, apparatus, device, medium, and program for quantizing a visual language model. The method includes: quantizing the visual modules of the visual language model during an offline quantization stage; compensating for quantization errors in the visual modules using a quantization compensator of the visual language model; and performing staged quantization processing on the language modules of the visual language model. The technical solution of this invention can improve the quantization accuracy of large-scale visual language models, thereby improving the inference efficiency of large-scale visual language models and reducing the inference time and computational cost of large-scale visual language models.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, storage medium and program for quantifying a visual language model. Background Technology

[0002] Large-scale visual-language models (VLMs) have achieved outstanding performance in multimodal reasoning tasks across various applications, including visual question answering and robot navigation. However, the massive computational and storage requirements of VLMs during inference pose significant challenges to practical engineering deployment. Linear layers, as one of the most common operations in VLMs, involve large matrix operations and their weights consume substantial amounts of storage, resulting in significant time and computational costs and becoming a major bottleneck for VLM deployment and inference.

[0003] Large model quantization refers to the method of converting model parameters from high precision (e.g., FP32, 4 bytes) to low precision (e.g., INT8, 1 byte). Within an acceptable range of precision loss, it can significantly reduce the resource consumption of large models and improve inference efficiency, making them more suitable for edge devices, mobile devices, or high-concurrency deployment scenarios. Currently, large visual language models often employ unified quantization methods, such as W4A16, W8A8, or W4A8 quantization.

[0004] In the process of developing this invention, the inventors discovered the following shortcomings in the existing technology: Large visual language models are typically composed of multiple different functional modules, each processing different data types, and each module also focuses on processing different data types at different stages. Applying a uniform quantization method to the entire large visual language model often results in low quantization accuracy, making it difficult to achieve the desired quantization effect. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, storage medium, and program for quantizing a visual language model, which can improve the quantization accuracy of large visual language models, thereby improving the inference efficiency of large visual language models and reducing the inference time and inference computation cost of large visual language models.

[0006] According to one aspect of the present invention, a method for quantizing a visual language model is provided, comprising: In the offline quantization stage of the model, the visual modules of the visual language model are quantized. The quantization error of the visual module is compensated by the quantization compensator of the visual language model. The language modules of the visual language model are subjected to phased quantization processing.

[0007] According to another aspect of the present invention, a quantization apparatus for a visual language model is provided, comprising: The visual module quantization processing module is used to quantize the visual modules of the visual language model during the offline quantization stage of the model. The quantization error compensation module is used to perform quantization compensation on the quantization error of the visual module through the quantization compensator of the visual language model. The language module quantization processing module is used to perform staged quantization processing on the language modules of the visual language model. According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the quantization method of the visual language model according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the quantization method of the visual language model according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the quantization method of the visual language model according to any embodiment of the present invention.

[0010] This invention addresses the problem of low quantization accuracy in existing visual language model quantization methods by quantizing the visual modules of a visual language model during the offline quantization stage, compensating for quantization errors in the visual modules using a quantization compensator for the visual language model, and then performing staged quantization on the language modules of the visual language model. This improves the quantization accuracy of large visual language models, thereby increasing the inference efficiency of large visual language models and reducing inference time and computational costs.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a method for quantizing a visual language model provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of a method for quantizing a visual language model provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a visual language model provided in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of an online reasoning process for a visual language model provided in Embodiment 2 of the present invention; Figure 5 This is a schematic diagram of a quantization device for a visual language model provided in Embodiment 3 of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1This is a flowchart of a quantization method for a visual language model provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where the functional modules of a visual language model are quantized separately. This method can be executed by a quantization device for the visual language model. This device can be implemented in software and / or hardware, and is generally integrated into an electronic device. This electronic device can be a terminal device or a server device, as long as it can execute the quantization method for the visual language model. The present invention does not limit the specific type of electronic device. Correspondingly, as... Figure 1 As shown, the method includes the following operations: S110. In the offline quantization stage of the model, the visual modules of the visual language model are quantized.

[0017] A visual language model can consist of a visual module and a language module. The visual module primarily processes image and video data, while the language module primarily processes text. The visual and language modules can be connected via a connection module. This connection module can exist independently of the visual and language modules, or it can be integrated within the language module; this embodiment of the invention does not impose any limitations on this.

[0018] The visual-language model can process multimodal data, which can be visual and text fusion data. Its basic data processing flow is as follows: First, acquire the multimodal data to be processed, which includes visual data (images or videos) and text data (questions, descriptions, or dialogues). Second, input the visual data into the visual module, which encodes the images or videos to extract corresponding visual feature sequences. Third, input the text data into the language module, which performs word segmentation and semantic encoding to extract corresponding text feature sequences. Fourth, input the visual feature sequences into the connection module, which maps and transforms them so that the transformed visual and text features are in a unified semantic feature space. When the connection module is independent of both the visual and language modules, the feature mapping is performed independently; when the connection module is integrated within the language module, the feature mapping is performed by the integrated connection structure within the language module. Finally, input the aligned visual and text features into the language module, which performs joint context modeling and semantic interaction to complete multimodal understanding and reasoning. Based on the multimodal joint inference results, the language module generates corresponding text results through autoregression and outputs these text results, including image question-answering answers, image description text, video understanding results, or multi-turn dialogue responses. Therefore, the visual language model can be a large-scale multimodal model.

[0019] In this embodiment of the invention, the visual language model can be first quantized offline during the offline quantization stage. The offline quantization stage is the stage where the model is quantized offline before being deployed online. During offline quantization, the visual modules of the visual language model can be quantized first.

[0020] In an optional embodiment of the present invention, the quantization processing of the visual modules of the visual language model may include: quantizing the weights of the linear layers of the visual modules sequentially based on the following formula; in, This represents the quantized weights of the visual module, and `clip()` represents the quantization function, which can specifically be a truncation function. Indicates the input array. The rounding symbol is used to round to the nearest integer. Indicates the lower bound of an element. This represents the upper limit of the element. W represents the weight of the visual module before quantization. This represents the scaling function for the weights, where A represents the quantization method, and N represents the quantization value under quantization method A. Optionally, the value of N will vary depending on the quantization method A. For example, when the quantization method A is 4, the value of N is 4.

[0021] For example, when performing offline quantization on the visual module, assume that the dimension of the weight W of the visual module is... ,in, Indicates the number of rows. Indicating the number of columns allows us to assign the weight W of the visual module along... The dimensions are quantized using N-bit methods, such as quantizing the weight W of the visual module using the following formula: in, This indicates that the weight W of the visual module is quantized using 4 bits. This represents the scaling function for the 4-bit quantization method. The quantization operation is repeated sequentially for each linear layer of the vision module until all linear layers in the vision module have been quantized.

[0022] S120. The quantization error of the visual module is compensated by the quantization compensator of the visual language model.

[0023] The quantization compensator can be a network structure module that simultaneously performs quantization and quantization error compensation. Quantization error in a model refers to the information loss or approximation error introduced during the conversion of parameters (such as weights and / or biases) or activation values ​​in a deep learning model from high precision (such as 32-bit floating-point numbers FP32) to low precision (such as 8-bit integers INT8) due to the discretization of numerical representation.

[0024] Since quantizing the visual module often causes a large overall quantization error, in order to eliminate the overall quantization error caused by quantizing the visual module, the structure of the visual language model can be improved by adding a learnable quantization compensator between the visual module and the language module of the visual language model.

[0025] A quantization compensator can compensate for quantization errors in the vision module. Quantization error compensation refers to the correction or compensation methods adopted to reduce the quantization errors introduced after quantizing model parameters (mapping continuous or high-precision values ​​to finite discrete values). Quantization compensation specifically refers to correcting and compensating for quantization errors caused by quantizing the vision module itself; that is, after compensating for the quantization errors of the vision module, the compensation result is output in a quantized form. Therefore, the result of quantization compensation for the quantization errors of the vision module through a quantization compensator is itself a quantization result. Optionally, the linear layers in the quantization compensator can be quantized using int8.

[0026] Since the quantization compensator can compensate for the overall quantization error caused by the quantization of the vision module, the vision module can use a low-bit quantization method with a larger quantization magnitude (the larger the quantization magnitude, the lower the quantization accuracy) during quantization. Furthermore, because the compensation result output by the quantization compensator is itself a quantized compensation result, this dual quantization processing method of "low-bit quantization + quantization error compensation" can, compared to the existing method of only performing conventional quantization on the vision module, further effectively improve the inference speed of the visual language model while ensuring quantization accuracy.

[0027] S130. Perform phased quantization processing on the language modules of the visual language model.

[0028] Understandably, the language module of a visual language model primarily consists of two stages during inference: the Prefill stage and the Decode stage. These two stages together constitute the complete process from understanding the input to generating the output. The Prefill stage of the language module mainly involves processing the entire input Prompt in parallel, calculating the Key / Value (KV) vectors of all Tokens (the smallest units into which the text is divided), and initializing the KV Cache. Because the Prefill stage involves large-scale matrix multiplication and self-attention calculations, its arithmetic intensity is high, making GPU (Graphics Processing Unit) computing power a limiting factor; therefore, the Prefill stage is a computationally intensive data processing stage. The Decode stage of the language module mainly involves autoregressively generating new Tokens one by one, calculating only the latest Token at each step, and reusing / updating the KV Cache. Since each Token generated in the Decode stage requires loading the entire growing KV Cache from GPU memory, the data transfer volume far exceeds the actual computation volume, resulting in low arithmetic intensity; therefore, the Decode stage is a memory-bandwidth-constrained data processing stage.

[0029] This shows that the bottlenecks of the language module in the visual language model vary at different data processing stages. Therefore, different quantization methods can be used for staged quantization processing at different data processing stages of the language module to adapt to the performance requirements of each data processing stage. This allows for personalized quantization processing of each data processing stage of the language module without affecting its data processing performance, thereby further improving the quantization accuracy of the language module.

[0030] Quantizing the visual language model using the aforementioned quantization method significantly reduces the computational cost of matrix multiplication, resulting in a marked reduction in forward inference time for single image-text pairs, image retrieval, or image-text question-answering tasks. The latency reduction is particularly pronounced in scenarios involving continuous multi-frame visual input and long text dialogues, supporting higher-frequency real-time interactions. Simultaneously, low-bit quantization directly reduces the storage overhead of visual language model weights and intermediate feature maps, allowing a single card or device to load larger batches of multimodal data simultaneously, or to perform large-scale model inference on resource-constrained devices that were previously impossible to run. In scenarios such as batch image-text retrieval, multimodal classification, and video frame-level understanding, the number of samples processed per unit time is significantly increased, optimizing overall system throughput. After visual language model quantization, floating-point operations in cross-modal interactions are converted to integer operations, reducing memory access pressure and computational latency, making multimodal feature alignment and fusion processes more efficient, especially for long texts and high-resolution images. Traditional floating-point visual language models are difficult to run on edge devices such as mobile phones, cameras, and robots. After quantization, visual language models are smaller and computationally lighter, allowing for direct deployment on the device for local multimodal inference. This avoids frequent transmission of image and text data to the cloud, reducing communication latency and enhancing privacy and security, thus enabling efficient multimodal understanding on the device. In scenarios requiring long-sequence inference, such as image and text generation, multi-turn dialogue, image narration, and long video understanding, model inference involves numerous steps and high intermediate state overhead. Visual language model quantization can stably reduce the inference overhead at each step, resulting in faster overall task pipeline execution, more stable resource usage, and prevention of inference interruptions due to memory overflow, thereby improving the execution efficiency and stability of complex tasks. In summary, quantization of visual language models can significantly improve inference speed and device throughput in key areas such as visual feature extraction, text semantic encoding, and cross-modal feature fusion by reducing computational precision and storage and memory access overhead. It also lowers the deployment threshold on the device, achieving more efficient and lightweight multimodal data inference while ensuring minimal loss of multimodal semantic understanding and alignment capabilities.

[0031] This invention addresses the problem of low quantization accuracy in existing visual language model quantization methods by quantizing the visual modules of a visual language model during the offline quantization stage, compensating for quantization errors in the visual modules using a quantization compensator for the visual language model, and then performing staged quantization on the language modules of the visual language model. This improves the quantization accuracy of large visual language models, thereby increasing the inference efficiency of large visual language models and reducing inference time and computational costs.

[0032] Example 2 Figure 2This is a flowchart of a quantization method for a visual language model provided in Embodiment 2 of the present invention. This embodiment is based on the above embodiment and is further specified. In this embodiment, various specific optional implementation methods are given, including compensating for the quantization error of the visual module through a quantization compensator of the visual language model, performing staged quantization processing on the language module of the visual language model, and the quantization method used in the online quantization stage of the model. Correspondingly, as Figure 2 As shown, the method in this embodiment may include: S210. In the offline quantization stage of the model, the visual modules of the visual language model are quantized.

[0033] S220. The quantization error of the visual module is compensated by the quantization compensator of the visual language model.

[0034] Figure 3 This is a schematic diagram of the structure of a visual language model provided in Embodiment 2 of the present invention. In an optional embodiment of the present invention, such as... Figure 3 As shown, the quantization compensator QAdapter (where Q represents the trained layer and Adapter represents the quantization compensator) can be composed of a first linear layer QLinear1, a modified activation function, and a second linear layer QLinear2. Optionally, the modified activation function can be a ReLU (Rectified Linear Unit) function. Correspondingly, quantization compensation for the quantization error of the visual module using the quantization compensator of the visual language model can include: quantizing the quantization error of the visual module based on the following formula: in, This represents the output after quantization compensation of the quantization error of the vision module, where α represents the rotation ratio of the quantization compensator. This represents the quantization compensation function of the first linear layer. This represents the quantization compensation function of the second linear layer. This represents the modified activation function. This represents the output of the vision module after quantization.

[0035] In the structure of the quantization compensator described above, the first linear layer QLinear1 can be used to initially compensate for the quantization error of the visual module. The modified activation function ReLU can be used to introduce nonlinearity to judge the compensation effect of the first linear layer QLinear1, enabling the model to learn and express complex patterns. The second linear layer QLinear2 can provide further quantization compensation based on the judgment result of the ReLU function and the quantization compensation result of the first linear layer QLinear1. Optionally, the first linear layer QLinear1 and the second linear layer QLinear2 can be used to perform 8-bit quantization on the activation and weights of the linear layers of the visual module. That is, the quantization compensator can further quantize and compensate the quantization result of the visual module into W8A8 form. It can be seen that through multi-level hierarchical quantization compensation processing, the quantization compensation accuracy of the quantization compensator can be effectively improved, thereby minimizing the quantization error of the visual module. At the same time, the output of the quantization compensator is the quantized compensation result, which, compared with the unquantized compensation method, can further improve the quantization accuracy and quantization effect of the visual language model, thereby further improving the inference speed of the visual language model.

[0036] In an optional embodiment of the present invention, before the quantization error of the visual module is compensated by the quantization compensator of the visual language model, the method may further include: using the visual module as a teacher model and the quantization compensator as a student model, and performing quantization training on the quantization compensator using a knowledge distillation strategy; wherein the loss function of the quantization compensator may be: Where Loss represents the loss value of the quantization compensator, M represents the row of the output matrix, and P represents the column of the output matrix. This indicates the unquantized output value of the vision module. This represents the output after quantization compensation of the quantization error of the vision module.

[0037] To efficiently learn the parameter information corresponding to the first and second linear layers in the quantization compensator, embodiments of this invention can employ a knowledge distillation strategy to pre-train the quantization compensator. Knowledge distillation is a model compression and transfer learning technique. Its core idea is to transfer the "knowledge" contained in a large and high-performance teacher model to a small and lightweight student model, thereby significantly reducing deployment costs and inference latency while maintaining high performance. Teacher models are typically complex and pre-trained, while student models have simpler structures and need to learn from the teacher. Optionally, the vision module can be used as the teacher model, and the quantization compensator as the student model. The knowledge distillation strategy can be used to train the quantization compensator using the model parameters of the vision module. Quantization training means that the quantization compensator is quantized during training, so that the trained quantization compensator itself is a quantized network structure.

[0038] S230. In the Prefill stage, the weights and activations of the language module are quantized using a first quantization method to obtain the first quantization parameters.

[0039] The first quantization method can be a quantization method that does not affect the computation speed. The first quantization parameter can be obtained by quantizing the weights and activations of the language modules using the first quantization method.

[0040] Specifically, in the offline quantization process of a visual language model, when the language module is in the Prefill stage, since the main bottleneck of the language module is computation, a first quantization method that will not significantly affect the computation speed can be used to quantize the weights and activations of the linear layer of the language module to obtain the first quantization parameters. For example, the first quantization method can be W8A8 quantization, where both activations and weights are quantized using 8 bits; matrix multiplication using int8 bits will improve computational efficiency.

[0041] S240. In the Decode stage, the weights of the language modules are quantized using a second quantization method to obtain second quantization parameters.

[0042] The second quantization method can be one that does not affect storage and bandwidth. The second quantization parameter can be obtained by quantizing the weights of the language modules using the second quantization method. The first quantization method has higher quantization precision than the second quantization method.

[0043] Specifically, during the offline quantization process of a visual language model, when the language module is in the Decode stage, since the main bottleneck of the language module is memory bandwidth, a second quantization method that will not significantly impact memory bandwidth and storage can be used to quantize the weights of the linear layers of the language module, obtaining the second quantization parameters. For example, the second quantization method can be W4A16 quantization, where the weights are quantized using 4 bits. These 4-bit weights only occupy one-quarter of the original model's storage, which will greatly alleviate storage and bandwidth pressure.

[0044] It is understandable that the quantization process of the visual module in the above visual language model, the process of the quantization compensator to compensate for the quantization error of the visual module, and the process of performing phased quantization processing on the language module all belong to the quantization strategy in the offline stage of the model.

[0045] S250. In the online quantization stage of the model, the offline quantization parameters of the model are loaded for online quantization processing.

[0046] In this embodiment of the invention, the visual language model can be quantized not only in the offline stage but also in the online stage.

[0047] In an optional embodiment of the present invention, the online quantization process of loading offline quantization parameters for online quantization during the online quantization stage may include: loading the offline quantization parameters obtained by quantizing the visual language model during the offline quantization stage, and generating a basic inference environment for the model based on the visual language model and the offline quantization parameters; quantizing the activation of the visual language model; loading the quantization parameters of the visual module into the visual module, and performing inference on the input image data through the visual module; loading the quantization parameters of the quantization compensator into the quantization compensator, and performing quantization compensation for the quantization error of the inference data of the visual module through the quantization compensator; in the Prefill stage, loading the first quantization parameters into the language module for forward inference; and in the Decode stage, loading the second quantization parameters into the language module for forward inference.

[0048] Among them, the offline quantization parameters can be the parameters obtained by the visual language model through quantization processing using multiple quantization methods during the offline quantization stage of the model.

[0049] Figure 4 This is a flowchart illustrating an online reasoning process for a visual language model according to Embodiment 2 of the present invention. In a specific example, such as... Figure 4As shown, in the online quantization phase of the visual language model, the offline quantization parameters obtained from the offline quantization phase can be loaded through the Runtime inference module. Based on the visual language model and the offline quantization parameters, the basic inference environment for generating the model is constructed. The Runtime inference module is responsible for loading, scheduling, and executing model computations; it is a key component in transitioning the model from the training state to actual service. The Runtime inference module directly manages the model's execution flow on hardware such as CPUs (Central Processing Units), GPUs, and NPUs (Neural Processing Units), ensuring efficient and low-latency inference.

[0050] After constructing the basic inference environment for the generative model, the activation of the visual language model can be quantized first. For example, the activation of the visual language model can be quantized... (N1 and) (Indicates the row and column where X is active), along The dimension is quantized using N bits. Considering the actual business accuracy requirements, N can be 4. That is, the activation of the visual language model can be quantized based on the following formula: in, This represents the activation value obtained after 4-bit quantization of the activation of the visual language model, where X represents the activation of the visual language model. This represents the scaling function for the activated 4-bit quantization method, where N can be 4.

[0051] Furthermore, the quantization parameters in W4A4 format used during offline quantization are loaded into the vision module; that is, the int4_GEMM calculation module is loaded into the vision module. This allows the vision module to perform inference on the input image data and efficiently perform matrix multiplication operations. For example, ... Figure 4 As shown, in the W4A4 quantization method, X represents the unquantized activation input, typically a bf16 (Brain Floating Point 16) data type. `int4_quant` indicates that it undergoes dynamic int4 quantization to obtain the quantized input. and its scale (scaling factor) value Load the pre-quantized 4-bit weights. ,Will and Perform 4-bit matrix multiplication and then multiply the result. This yields a W4A4 quantized output of a Linear. The computation process for other W4A4 quantized Linears in the visual language model follows the same logic. Correspondingly, the quantized output of the online inference process of the vision module can be: Where O represents the quantization result output by the online inference process of the vision module. The scaling factor represents the 4-bit quantization method used to represent the weights. This represents the result of 4-bit quantization of the input image data. This indicates the weights quantized using the 4-bit quantization method.

[0052] Furthermore, the quantization parameters of the quantization compensator are loaded into the quantization compensator, and the quantization error of the inference data from the vision module is compensated by the quantization compensator for forward inference. In the Prefill stage, the first quantization parameter can be loaded into the language module for forward inference; in the Decode stage, the second quantization parameter can be loaded into the language module for forward inference.

[0053] Optionally, the quantization compensator can employ a W8A8 quantization method to compensate for quantization errors in the inference data of the vision module during forward inference. In the Prefill stage, relevant W8A8 parameters can be loaded into the language module for forward inference. In the Decode stage, relevant W4A16 parameters can be loaded into the language module for forward inference.

[0054] For example, such as Figure 4 As shown, in the W8A8 quantization method, X represents the unquantized activation input (usually a bf16 data type), and int8_quant indicates that it is dynamically quantized to int8 to obtain the quantized input. and its scaling factor (scale value) Load the pre-quantized 8-bit weights. ,Will and Multiply the result of the 8-bit matrix multiplication. This yields a Linear output after W8A8 quantization. The computation process for other Linear values ​​in the visual language model using W8A8 quantization follows the same logic. In W4A16 quantization, pre-quantized 4-bit weights can be loaded. and its scale value The fp16_dequant operation can be implemented by running... Dequantize the 4-bit weights into an fp16 dataset type. Activate the unquantized input. with the dequantized weights Matrix multiplication will produce an output of type fp16. The computation process for other linear data using W4A16 quantization in the visual language model follows the same logic. The scaling factor represents the 8-bit quantization method used to represent the weights.

[0055] In summary, the above technical solution provides a low-precision quantization method for visual language models. This quantization method includes two parts: offline quantization and online inference. In the offline quantization calculation part, the activations and weights in the visual module are first quantized using N-bit quantization. Then, the quantization error caused by the N-bit quantization is compensated by the quantization compensator QAdapter configured after the visual module. Finally, the language module is quantized in stages: W8A8 quantization is performed when the language module is in the Prefill stage, and W4A16 quantization is performed when the language module is in the Decode stage. In the online inference part, the Runtime inference module is first used to load fixed quantization parameters to build a basic inference environment. Then, the activations of the visual module are quantized using N-bit quantization, and the quantized N-bit weights of the visual module are loaded. Then, low-bit matrix multiplication is used for fast operation. Next, the relevant parameters of the quantization compensator QAdapter are loaded for forward inference. Finally, the language module is inferred in stages. The above quantization method for visual language models has low precision loss and fast computation speed, which can effectively improve the inference efficiency of visual language models and reduce the inference time and computation cost of visual language models.

[0056] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant regions.

[0057] It should be noted that any arrangement or combination of the technical features in the above embodiments also falls within the protection scope of this invention.

[0058] Example 3 Figure 5 This is a schematic diagram of a quantization device for a visual language model provided in Embodiment 3 of the present invention, as shown below. Figure 5 As shown, the device includes: a vision module quantization processing module 510, a quantization error compensation module 520, and a language module quantization processing module 530, wherein: The visual module quantization processing module 510 is used to quantize the visual modules of the visual language model during the offline quantization stage of the model. The quantization error compensation module 520 is used to perform quantization compensation on the quantization error of the visual module through the quantization compensator of the visual language model. The language module quantization processing module 530 is used to perform phased quantization processing on the language modules of the visual language model.

[0059] This invention addresses the problem of low quantization accuracy in existing visual language model quantization methods by quantizing the visual modules of a visual language model during the offline quantization stage, compensating for quantization errors in the visual modules using a quantization compensator for the visual language model, and then performing staged quantization on the language modules of the visual language model. This improves the quantization accuracy of large visual language models, thereby increasing the inference efficiency of large visual language models and reducing inference time and computational costs.

[0060] Optionally, the visual module quantization processing module 510 is further configured to: quantize the weights of the linear layer of the visual module sequentially based on the following formula; in, The weights of the visual module after quantization are represented by 'clip()', where 'clip()' represents the quantization function, and 'W' represents the weights of the visual module before quantization. The scaling function represents the weights, A represents the quantization method, and N represents the quantization value under quantization method A.

[0061] Optionally, the quantization compensator consists of a first linear layer, a modified activation function, and a second linear layer; the quantization error compensation module 520 is further configured to: perform quantization compensation on the quantization error of the vision module using the quantization compensator based on the following formula: in, This represents the output after quantization compensation of the quantization error of the vision module, where α represents the rotation ratio of the quantization compensator. This represents the quantization compensation function of the first linear layer. This represents the quantization compensation function of the second linear layer. This represents the modified activation function. This represents the output of the vision module after quantization.

[0062] Optionally, the above device further includes a quantization compensator quantization training module, used to: use the visual module as a teacher model, use the quantization compensator as a student model, and use a knowledge distillation strategy to quantize and train the quantization compensator. The loss function of the quantization compensator is: Where Loss represents the loss value of the quantization compensator, M represents the row of the output matrix, and P represents the column of the output matrix. This indicates the unquantized output value of the vision module.

[0063] Optionally, the language module quantization processing module 530 is further configured to: in the Prefill stage, quantize the weights and activations of the language module using a first quantization method to obtain a first quantization parameter; in the Decode stage, quantize the weights of the language module using a second quantization method to obtain a second quantization parameter; wherein the quantization precision of the first quantization method is higher than that of the second quantization method.

[0064] Optionally, the above apparatus further includes an online quantization module, configured to: in the online quantization stage, load the offline quantization parameters obtained by the visual language model during the offline quantization stage, and generate a basic inference environment for the model based on the visual language model and the offline quantization parameters; quantize the activation of the visual language model; load the quantization parameters of the visual module into the visual module, and perform inference on the input image data through the visual module; load the quantization parameters of the quantization compensator into the quantization compensator, and perform quantization compensation for the quantization error of the inference data of the visual module through the quantization compensator; in the Prefill stage, load the first quantization parameters into the language module for forward inference; and in the Decode stage, load the second quantization parameters into the language module for forward inference.

[0065] The quantization device for the visual language model described above can execute the quantization method for the visual language model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the quantization method for the visual language model provided in any embodiment of the present invention.

[0066] Since the visual language model quantization device described above is an apparatus capable of executing the visual language model quantization method in the embodiments of the present invention, those skilled in the art can understand the specific implementation and various variations of the visual language model quantization device in this embodiment based on the visual language model quantization method described in the embodiments of the present invention. Therefore, how the visual language model quantization device implements the visual language model quantization method in the embodiments of the present invention will not be described in detail here. Any apparatus used by those skilled in the art to implement the visual language model quantization method in the embodiments of the present invention falls within the scope of protection of this application.

[0067] Example 4 Figure 6A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0068] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0069] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0070] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the quantization method of a visual language model.

[0071] Optionally, the quantization method for the visual language model may include: quantizing the visual modules of the visual language model during the offline quantization stage; quantizing the quantization error of the visual modules using the quantization compensator of the visual language model; and performing staged quantization processing on the language modules of the visual language model.

[0072] In some embodiments, the method for quantizing the visual language model can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or mounted on the electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for quantizing the visual language model described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the method for quantizing the visual language model by any other suitable means (e.g., by means of firmware).

[0073] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0074] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0075] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0076] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0077] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0078] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0079] This invention also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the visual language model quantization method provided in any embodiment of this invention. This program product shares the same inventive concept as the visual language model quantization methods disclosed in the embodiments of this invention, and therefore will not be described further here.

[0080] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0081] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A quantization method for a visual language model, characterized in that, include: In the offline quantization stage of the model, the visual module of the visual language model is quantized; wherein, the visual module of the visual language model is used to process image and video data; The quantization error of the visual module is compensated by the quantization compensator of the visual language model. The language module of the visual language model is subjected to staged quantization processing; wherein, the language module of the visual language model is used to process text; Before the quantization compensation of the quantization error of the visual module is performed by the quantization compensator of the visual language model, the method further includes: The visual module is used as the teacher model, the quantization compensator is used as the student model, and the quantization compensator is trained using a knowledge distillation strategy. The loss function of the quantization compensator is: Where Loss represents the loss value of the quantization compensator, M represents the row of the output matrix, and P represents the column of the output matrix. This represents the output after quantization compensation of the quantization error of the vision module. This indicates the unquantized output value of the vision module; The step of performing phased quantization processing on the language modules of the visual language model includes: In the prefill stage, the weights and activations of the language modules are quantized using a first quantization method to obtain first quantization parameters; During the decoding stage, the weights of the language modules are quantized using a second quantization method to obtain second quantization parameters; The quantization accuracy of the first quantization method is higher than that of the second quantization method.

2. The method according to claim 1, characterized in that, The quantization process of the visual modules of the visual language model includes: The weights of the linear layer of the vision module are quantized sequentially based on the following formula; in, The weights of the visual module after quantization are represented by 'clip()', where 'clip()' represents the quantization function, and 'W' represents the weights of the visual module before quantization. The scaling function represents the weights, A represents the quantization method, and N represents the quantization value under quantization method A.

3. The method according to claim 1, characterized in that, The quantization compensator consists of a first linear layer, a modified activation function, and a second linear layer; the quantization compensation of the quantization error of the visual module through the quantization compensator of the visual language model includes: The quantization error of the vision module is compensated by the quantization compensator based on the following formula: in, This represents the output after quantization compensation of the quantization error of the vision module, where α represents the rotation ratio of the quantization compensator. This represents the quantization compensation function of the first linear layer. This represents the quantization compensation function of the second linear layer. This represents the modified activation function. This represents the output of the vision module after quantization.

4. The method according to claim 1, characterized in that, Also includes: During the online quantization phase of the model, the offline quantization parameters obtained by the visual language model during the offline quantization phase are loaded, and a basic inference environment for the model is generated based on the visual language model and the offline quantization parameters. The activation of the visual language model is quantized; The quantization parameters of the vision module are loaded into the vision module, and the input image data is inferred through the vision module. The quantization parameters of the quantization compensator are loaded into the quantization compensator, and the quantization error of the inference data of the vision module is compensated by the quantization compensator. During the Prefill phase, the first quantization parameter is loaded into the language module for forward inference; During the Decode phase, the second quantization parameter is loaded into the language module for forward inference.

5. A quantization device for a visual language model, characterized in that, include: The visual module quantization processing module is used to quantize the visual modules of the visual language model during the offline quantization stage of the model; wherein, the visual modules of the visual language model are used to process image and video data. The quantization error compensation module is used to perform quantization compensation on the quantization error of the visual module through the quantization compensator of the visual language model. The language module quantization processing module is used to perform staged quantization processing on the language modules of the visual language model; wherein, the language modules of the visual language model are used to process text. The quantization compensator quantization training module is used to: use the vision module as a teacher model, use the quantization compensator as a student model, and use a knowledge distillation strategy to quantize and train the quantization compensator. The loss function of the quantization compensator is: Where Loss represents the loss value of the quantization compensator, M represents the row of the output matrix, and P represents the column of the output matrix. This represents the output after quantization compensation of the quantization error of the vision module. This indicates the unquantized output value of the vision module; The language module quantization processing module is further configured to: in the Prefill stage, quantize the weights and activations of the language module using a first quantization method to obtain a first quantization parameter; in the Decode stage, quantize the weights of the language module using a second quantization method to obtain a second quantization parameter; wherein the quantization precision of the first quantization method is higher than that of the second quantization method.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that is executed by the at least one processor to enable the at least one processor to perform the quantization method of the visual language model according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the quantization method of the visual language model according to any one of claims 1-4.

8. A computer program product, characterized in that, Includes a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the quantization method of the visual language model according to any one of claims 1-4.

Images