Model quantification method, model reasoning method, device, equipment, medium and product

Abstract

The invention provides a model quantification method and device, a model reasoning method and device, equipment, a medium and a product. Performing sensitivity evaluation on each network channel based on the weight matrix to obtain the channel sensitivity of each network channel; based on the channel sensitivity of the network channel corresponding to each element in the weight matrix, rearranging and dividing each element in the weight matrix to obtain a plurality of weight sub-matrixes; based on the channel sensitivity of the network channel corresponding to each element in each weight sub-matrix, allocating quantitative resources to each weight sub-matrix; performing quantization processing on the weight sub-matrixes based on the quantization resources to obtain a plurality of quantization sub-matrixes; and correcting the plurality of quantization sub-matrixes to obtain a target quantization matrix. Elements in a weight matrix can be accurately and orderly gathered according to channel sensitivity, different weight parameters are finely classified, different weight sub-matrixes are compatible for quantification, the problem of weight heterogeneity caused by multiple modes is solved, and the accuracy of model quantification is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a model quantization method, a model inference method, an apparatus, a device, a medium, and a product. Background Technology

[0002] Visual-language models (VLMs) have made significant progress in cross-modal understanding and reasoning, but their extremely large parameter size leads to huge memory and bandwidth overhead during deployment. Post-training quantization (PTQ) has become the mainstream model compression scheme because it eliminates the need for retraining, and thus, quantization methods for visual-language models have emerged.

[0003] In related technologies, quantization methods applicable to various models are typically used to quantize visual language models. However, because visual language models support the processing of multimodal data, the weight parameters exhibit weight heterogeneity caused by multimodality. That is, the statistical characteristics of images and text tokens (the basic units of input models, such as words, characters, or sub-words) differ greatly. General quantization methods struggle to take into account the heterogeneity between different weight parameters, resulting in low accuracy in model quantization. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a model quantization method, a model inference method, an apparatus, a device, a medium, and a product.

[0005] On one hand, embodiments of this disclosure provide a model quantization method, including:

[0006] Based on the weight matrix of the visual language model, the sensitivity of each network channel of the visual language model is evaluated to obtain the channel sensitivity of each network channel. The weight matrix includes elements corresponding to multiple network channels.

[0007] Based on the channel sensitivity of the network channels corresponding to each element in the weight matrix, the elements in the weight matrix are rearranged and divided to obtain multiple weight submatrices.

[0008] Based on the channel sensitivity of the network channel corresponding to each element in each weight submatrix, corresponding quantization resources are allocated to each weight submatrix.

[0009] Based on the quantization resources corresponding to each weight submatrix, each weight submatrix is ​​quantized to obtain multiple quantized submatrixes.

[0010] The plurality of quantization sub-matrices are modified to obtain the target quantization matrix of the weight matrix, so that the visual language model can be deployed to a hardware device to perform visual language tasks based on the target quantization matrix.

[0011] In one possible implementation, the weight matrix based on the visual language model is used to evaluate the sensitivity of each network channel of the visual language model, obtaining the channel sensitivity of each network channel, including:

[0012] Based on the weight matrix and the loss function of the visual language model, the global sensitivity of the weight matrix for each network channel is evaluated to obtain the global sensitivity matrix corresponding to the weight matrix.

[0013] The global sensitivity of each network channel is obtained based on the global sensitivity matrix.

[0014] Based on the weight matrix and the activation sample set of the visual language model, the local sensitivity of the weight matrix for each network channel is evaluated to obtain the local sensitivity of each network channel.

[0015] The channel sensitivity of each network channel is obtained based on the global sensitivity and local sensitivity of each network channel.

[0016] In one possible implementation, each network channel includes an input channel and an output channel; the global sensitivity matrix includes a first global matrix and a second global matrix, wherein the first global matrix represents the global sensitivity of the weight matrix to each input channel, and the second global matrix represents the global sensitivity of the weight matrix to each output channel.

[0017] The loss function based on the weight matrix and the visual language model is used to evaluate the global sensitivity of the weight matrix for each network channel, resulting in a global sensitivity matrix corresponding to the weight matrix, including any one of the following:

[0018] Based on the gradient of the loss function of the visual language model with respect to the weight matrix, determine the first global matrix and the second global matrix corresponding to the weight matrix;

[0019] Determine the second-order partial derivative matrix of the loss function of the visual language model with respect to the weight matrix, and decompose the second-order partial derivative matrix to obtain the first global matrix and the second global matrix corresponding to the weight matrix;

[0020] Accordingly, obtaining the global sensitivity of each network channel based on the global sensitivity matrix includes:

[0021] The global sensitivity of each input channel is obtained based on the diagonal elements of the first global matrix, and the global sensitivity of each output channel is obtained based on the diagonal elements of the second global matrix.

[0022] In one possible implementation, the evaluation of the local sensitivity of the weight matrix for each network channel based on the weight matrix and the activation sample set of the visual language model to obtain the local sensitivity of each network channel includes:

[0023] Obtain the sample activation matrix corresponding to the activated sample set;

[0024] Based on the elements corresponding to each input channel in the sample activation matrix and the weight matrix, the local sensitivity of the weight matrix to each input channel is determined;

[0025] Based on the elements corresponding to each output channel in the sample activation matrix and the weight matrix, the local sensitivity of the weight matrix to each output channel is determined.

[0026] In one possible implementation, the step of modifying the plurality of quantization sub-matrices to obtain the target quantization matrix of the weight matrix includes:

[0027] The initial quantization matrix of the visual language model is obtained based on the multiple quantization sub-matrices;

[0028] Matrix decomposition is performed on the first global matrix and the second global matrix corresponding to the weight matrix to obtain the first lower triangular matrix and the first diagonal matrix corresponding to the first global matrix, and the second lower triangular matrix and the second diagonal matrix corresponding to the second global matrix.

[0029] Based on the first lower triangular matrix, the second lower triangular matrix, the first diagonal matrix, and the second diagonal matrix, the initial quantization matrix is ​​modified to obtain the target quantization matrix.

[0030] In one possible implementation, the step of modifying the initial quantization matrix based on the first lower triangular matrix, the second lower triangular matrix, the first diagonal matrix, and the second diagonal matrix to obtain the target quantization matrix includes:

[0031] The following steps are performed iteratively until the convergence condition is met, at which point the iteration stops, and the iterative quantization matrix obtained in the last iteration is used as the target quantization matrix:

[0032] The iterative quantization matrix obtained in the previous iteration is used as the quantization matrix to be corrected in the current iteration, and the quantization matrix to be corrected in the first iteration is used as the initial quantization matrix.

[0033] Determine the residual between the weight matrix and the quantization matrix to be corrected;

[0034] Based on the residual, the first lower triangular matrix, and the second lower triangular matrix, the first-order compensation term is determined;

[0035] The first lower triangular matrix and the second lower triangular matrix are inversely transformed to obtain the intermediate matrix. The correction coefficients are determined based on the first diagonal matrix and the second diagonal matrix, and the interaction compensation term is obtained based on the intermediate matrix and the correction coefficients.

[0036] Based on the residual, the first lower triangular matrix, and the second lower triangular matrix, the second-order compensation term is determined;

[0037] The weight matrix is ​​modified based on the first-order compensation term, the interaction compensation term, and the second-order compensation term to obtain the modified matrix for this iteration.

[0038] The correction matrix of this iteration is quantized to obtain the iteration quantization matrix of this iteration;

[0039] The convergence condition includes that the difference between the iterative quantization matrix obtained in this iteration and the quantization matrix to be corrected in this iteration is not higher than a preset difference threshold.

[0040] In one possible implementation, the elements in the weight matrix are rearranged and partitioned based on the channel sensitivity of the network channels corresponding to each element in the weight matrix to obtain multiple weight sub-matrices, including:

[0041] Based on the channel sensitivity of the output channel corresponding to each element in the weight matrix, the rows in the weight matrix are sorted in descending order of channel sensitivity from largest to smallest. Also, based on the channel sensitivity of the input channel corresponding to each element in the weight matrix, the columns in the weight matrix are sorted in descending order of channel sensitivity from largest to smallest, resulting in a rearranged weight matrix.

[0042] The rearranged weight matrix is ​​divided into multiple blocks to obtain multiple weight submatrices.

[0043] In one possible implementation, allocating corresponding quantization resources to each weight submatrix based on the channel sensitivity of the network channel corresponding to each element in each weight submatrix includes:

[0044] For each weight submatrix, the saliency of each element in each weight submatrix is ​​determined based on the channel sensitivity of the input channel and the channel sensitivity of the output channel corresponding to each element in each weight submatrix.

[0045] The significance of each weight submatrix is ​​determined based on the significance of multiple elements in each weight submatrix;

[0046] Based on the significance of each weight submatrix, corresponding quantization resources are allocated to each weight submatrix.

[0047] In another aspect, embodiments of this disclosure provide a model reasoning method, including:

[0048] In response to receiving a visual language task, the visual language model is invoked to process the natural language instructions and visual information in the visual language task, and the task processing result is obtained.

[0049] The target quantization matrix obtained by the visual language model based on the above model quantization method is pre-deployed to the hardware device.

[0050] In another aspect, embodiments of this disclosure provide a model quantization apparatus, including:

[0051] An evaluation module is used to evaluate the sensitivity of each network channel of the visual language model based on the weight matrix of the visual language model, and obtain the channel sensitivity of each network channel. The weight matrix includes elements corresponding to multiple network channels.

[0052] The partitioning module is used to rearrange and partition the elements in the weight matrix based on the channel sensitivity of the network channels corresponding to each element in the weight matrix, so as to obtain multiple weight submatrices.

[0053] The allocation module is used to allocate corresponding quantization resources to each weight submatrix based on the channel sensitivity of the network channel corresponding to each element in each weight submatrix.

[0054] The determination module is used to quantize each weight submatrix based on the quantization resources corresponding to each weight submatrix, and obtain multiple quantized submatrixes.

[0055] The correction module is used to correct the plurality of quantization sub-matrices to obtain the target quantization matrix of the weight matrix, so as to deploy the visual language model to the hardware device to perform visual language tasks based on the target quantization matrix.

[0056] In one possible implementation, the evaluation module includes:

[0057] The first evaluation unit is used to evaluate the global sensitivity of the weight matrix for each network channel based on the weight matrix and the loss function of the visual language model, so as to obtain the global sensitivity matrix corresponding to the weight matrix.

[0058] The first determining unit is used to obtain the global sensitivity of each network channel based on the global sensitivity matrix;

[0059] The second evaluation unit is used to evaluate the local sensitivity of the weight matrix for each network channel based on the weight matrix and the activation sample set of the visual language model, so as to obtain the local sensitivity of each network channel.

[0060] The second determining unit is used to obtain the channel sensitivity of each network channel based on the global sensitivity and local sensitivity of each network channel.

[0061] In one possible implementation, each network channel includes an input channel and an output channel; the global sensitivity matrix includes a first global matrix and a second global matrix, wherein the first global matrix represents the global sensitivity of the weight matrix to each input channel, and the second global matrix represents the global sensitivity of the weight matrix to each output channel.

[0062] The first evaluation unit is used for any of the following:

[0063] Based on the gradient of the loss function of the visual language model with respect to the weight matrix, determine the first global matrix and the second global matrix corresponding to the weight matrix;

[0064] Determine the second-order partial derivative matrix of the loss function of the visual language model with respect to the weight matrix, and decompose the second-order partial derivative matrix to obtain the first global matrix and the second global matrix corresponding to the weight matrix;

[0065] Accordingly, the first determining unit is used to obtain the global sensitivity of each input channel based on the diagonal elements in the first global matrix, and to obtain the global sensitivity of each output channel based on the diagonal elements in the second global matrix.

[0066] In one possible implementation, the second evaluation unit is used for:

[0067] Obtain the sample activation matrix corresponding to the activated sample set;

[0068] Based on the elements corresponding to each input channel in the sample activation matrix and the weight matrix, the local sensitivity of the weight matrix to each input channel is determined;

[0069] Based on the elements corresponding to each output channel in the sample activation matrix and the weight matrix, the local sensitivity of the weight matrix to each output channel is determined.

[0070] In one possible implementation, the correction module includes:

[0071] The third determining unit is used to obtain the initial quantization matrix of the visual language model based on the plurality of quantization sub-matrices;

[0072] The decomposition unit is used to perform matrix decomposition on the first global matrix and the second global matrix corresponding to the weight matrix, respectively, to obtain the first lower triangular matrix and the first diagonal matrix corresponding to the first global matrix, and the second lower triangular matrix and the second diagonal matrix corresponding to the second global matrix.

[0073] The correction unit is used to correct the initial quantization matrix based on the first lower triangular matrix, the second lower triangular matrix, the first diagonal matrix, and the second diagonal matrix to obtain the target quantization matrix.

[0074] In one possible implementation, the correction unit, when correcting the initial quantization matrix based on the first lower triangular matrix, the second lower triangular matrix, the first diagonal matrix, and the second diagonal matrix to obtain the target quantization matrix, is specifically used for:

[0075] The following steps are performed iteratively until the convergence condition is met, at which point the iteration stops, and the iterative quantization matrix obtained in the last iteration is used as the target quantization matrix:

[0076] The iterative quantization matrix obtained in the previous iteration is used as the quantization matrix to be corrected in the current iteration, and the quantization matrix to be corrected in the first iteration is used as the initial quantization matrix.

[0077] Determine the residual between the weight matrix and the quantization matrix to be corrected;

[0078] Based on the residual, the first lower triangular matrix, and the second lower triangular matrix, the first-order compensation term is determined;

[0079] The first lower triangular matrix and the second lower triangular matrix are inversely transformed to obtain the intermediate matrix. The correction coefficients are determined based on the first diagonal matrix and the second diagonal matrix, and the interaction compensation term is obtained based on the intermediate matrix and the correction coefficients.

[0080] Based on the residual, the first lower triangular matrix, and the second lower triangular matrix, the second-order compensation term is determined;

[0081] The weight matrix is ​​modified based on the first-order compensation term, the interaction compensation term, and the second-order compensation term to obtain the modified matrix for this iteration.

[0082] The correction matrix of this iteration is quantized to obtain the iteration quantization matrix of this iteration;

[0083] The convergence condition includes that the difference between the iterative quantization matrix obtained in this iteration and the quantization matrix to be corrected in this iteration is not higher than a preset difference threshold.

[0084] In one possible implementation, the partitioning module is used for:

[0085] Based on the channel sensitivity of the output channel corresponding to each element in the weight matrix, the rows in the weight matrix are sorted in descending order of channel sensitivity from largest to smallest. Also, based on the channel sensitivity of the input channel corresponding to each element in the weight matrix, the columns in the weight matrix are sorted in descending order of channel sensitivity from largest to smallest, resulting in a rearranged weight matrix.

[0086] The rearranged weight matrix is ​​divided into multiple blocks to obtain multiple weight submatrices.

[0087] In one possible implementation, the allocation module is configured to:

[0088] For each weight submatrix, the saliency of each element in each weight submatrix is ​​determined based on the channel sensitivity of the input channel and the channel sensitivity of the output channel corresponding to each element in each weight submatrix.

[0089] The significance of each weight submatrix is ​​determined based on the significance of multiple elements in each weight submatrix;

[0090] Based on the significance of each weight submatrix, corresponding quantization resources are allocated to each weight submatrix.

[0091] In another aspect, embodiments of this disclosure provide a model inference apparatus, including:

[0092] The calling module is used to respond to a received visual language task by calling a visual language model to process the natural language instructions and visual information in the visual language task and obtain the task processing result.

[0093] The target quantization matrix obtained by the visual language model based on the above model quantization method is pre-deployed to the hardware device.

[0094] In another aspect, embodiments of this disclosure provide an electronic device, including: a memory for storing a computer program product; and a processor for executing the computer program product stored in the memory, wherein when the computer program product is executed, it implements the above-described model quantization method or model inference method.

[0095] In another aspect, embodiments of this disclosure provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the aforementioned model quantization method or model inference method.

[0096] In another aspect, embodiments of this disclosure provide a computer program product, including computer program instructions that, when executed by a processor, implement the aforementioned model quantization method or model inference method.

[0097] The model quantization method based on this disclosure evaluates the channel sensitivity of each network channel using a weight matrix based on a visual language model. This weight matrix includes elements corresponding to multiple network channels. Based on the channel sensitivity of the network channels corresponding to each element in the weight matrix, the elements in the weight matrix are rearranged and divided to obtain multiple weight sub-matrices. This ensures that the elements in the weight matrix are distributed in an ordered manner according to their degree of influence, with elements of similar influence grouped into the same weight sub-matrices, providing the possibility for subsequent differentiated quantization of resources for different weight sub-matrices. Then, based on the channel sensitivity of the network channels corresponding to each element in each weight sub-matrice, quantization resources are allocated to each weight sub-matrice. Each weight sub-matrice is quantized based on these quantization resources to obtain multiple quantized sub-matrices. Finally, these multiple quantized sub-matrices are corrected to obtain the target quantization matrix. Because the elements in the weight matrix are rearranged and partitioned, they are precisely and orderly clustered according to channel sensitivity, thereby finely classifying the degree of influence of different weight parameters on the model; and different resources are precisely allocated to different weight submatrices, and different weight submatrices are compatible for accurate quantization, so that quantization resources are concentrated in more important areas of the weight matrix, which can effectively overcome the weight heterogeneity problem caused by multimodality and improve the accuracy of model quantization.

[0098] The model quantization method based on this disclosure, upon receiving a visual language task, invokes a visual language model to process the natural language instructions and visual information in the task, obtaining the task processing result. Since the visual language model is based on a target quantization matrix obtained using the aforementioned model quantization method, which is pre-deployed on the hardware device and is compatible with different weight sub-matrices for accurate quantization, it overcomes the weight heterogeneity problem caused by multimodal processing. Therefore, performing the visual language task based on this target quantization matrix yields output results, improving the accuracy of model inference.

[0099] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0100] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0101] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0102] Figure 1A schematic flowchart of a model quantization method provided in an embodiment of this disclosure;

[0103] Figure 2 A schematic diagram of a preliminary quantization process based on channel sensitivity provided in an embodiment of this disclosure;

[0104] Figure 3 A schematic diagram of vector quantization provided for an embodiment of this disclosure;

[0105] Figure 4 This is a statistical diagram of channel characteristics based on multimodal analysis obtained from research on related technologies.

[0106] Figure 5 This is a schematic diagram illustrating the heterogeneity of weight distribution caused by multimodal processing, obtained from research on related technologies.

[0107] Figure 6 A flowchart illustrating another model quantization method provided in this embodiment of the disclosure;

[0108] Figure 7 A flowchart illustrating yet another model quantization method provided in this disclosure embodiment;

[0109] Figure 8 A flowchart illustrating another model quantization method provided in an embodiment of this disclosure;

[0110] Figure 9 A flowchart illustrating yet another model quantization method provided in this disclosure embodiment;

[0111] Figure 10 A flowchart illustrating yet another model quantization method provided in this disclosure embodiment;

[0112] Figure 11 A first-order gradient statistics diagram provided in this embodiment of the disclosure;

[0113] Figure 12 A flowchart illustrating yet another model quantization method provided in this disclosure embodiment;

[0114] Figure 13 A flowchart illustrating yet another model quantization method provided in this disclosure embodiment;

[0115] Figure 14 A flowchart illustrating a model reasoning method provided in an embodiment of this disclosure;

[0116] Figure 15 This is a schematic diagram of the structure of a model quantization device provided in an embodiment of the present disclosure;

[0117] Figure 16 This is a schematic diagram of the structure of a model inference device provided in an embodiment of the present disclosure;

[0118] Figure 17 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0119] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0120] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0121] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0122] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0123] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0124] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship. The data referred to in this disclosure can include unstructured data such as text, images, and videos, as well as structured data.

[0125] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0126] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0127] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0128] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0129] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0130] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0131] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0132] Figure 1 This is a schematic flowchart of a model quantization method provided in an embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 1 As shown, it includes the following steps:

[0133] Step 101: Based on the weight matrix of the visual language model, evaluate the sensitivity of each network channel of the visual language model to obtain the channel sensitivity of each network channel.

[0134] The weight matrix includes elements corresponding to multiple network channels, which include input and output channels.

[0135] Channel sensitivity characterizes the degree of influence of the weight matrix on the network channels. Higher sensitivity indicates a greater influence of the weight matrix on the network channels, meaning higher importance. In this step, the global sensitivity of each network channel is obtained by evaluating the global influence of each element in the weight matrix on the visual language model; and the local sensitivity of each network channel is obtained by evaluating the local influence of each element in the weight matrix on each network channel. Based on the global and local sensitivity of each network channel, the channel sensitivity of each network channel is obtained.

[0136] For example, to assess the global impact, the loss function of the weight matrix of the visual language model can be used to evaluate the influence of each element of the network channel in the weight matrix on the global loss. To assess the local impact, the activation sample set can be used to evaluate the influence of each element of the network channel in the weight matrix on the local activation. The activation sample set may include multiple samples, and the data types of the samples may include, but are not limited to, text, images, or audio / video data.

[0137] It should be noted that the Vision Language Model (VLM) has made significant progress in cross-modal understanding and reasoning. The VLM integrates computer vision (CV) and natural language processing (NLP) techniques, enabling simultaneous processing and understanding of images and text, achieving joint understanding and generation of multimodal information; for example, visual question answering tasks, image description generation, and multimodal dialogue. However, the extremely large parameter scale of the VLM leads to huge memory and bandwidth overhead during deployment. In this embodiment of the disclosure, the parameters of the VLM can be quantized, and these parameters may include a weight matrix.

[0138] Step 102: Based on the channel sensitivity of the network channel corresponding to each element in the weight matrix, rearrange and divide each element in the weight matrix to obtain multiple weight submatrices.

[0139] In this step, the elements in the weight matrix can be rearranged according to the sensitivity of the input and output channels corresponding to each element. This rearranges elements with similar sensitivity to network channels, placing them closer together. For example, elements with high sensitivity across all network channels are clustered together, while elements with significantly different sensitivity are placed further apart. The rearranged weight matrix can then be divided into multiple weight submatrices.

[0140] Step 101 can be called Channel Sensitivity Assessment (CSA), and step 102 can be called Matrix Reordering and Structured Block Partitioning (MRSBP).

[0141] like Figure 2 As shown, the channel sensitivity of each network channel is obtained through CSA. MRSBP is used to cluster high-sensitivity regions in the weight matrix. Specifically, as... Figure 2 As shown, through CSA, the elements corresponding to each network channel in the weight matrix have different levels of importance; the higher the importance of a network channel, the darker the color of the element corresponding to that network channel. After reordering by MRSBP, the elements are arranged in order according to the importance of their corresponding network channels. Figure 2 The data shows that elements of high importance are clustered in the left and upper parts of the weight matrix. For example, the weight matrix can be evenly divided into four blocks, b1, b2, b3, and b4, using MRSBP.

[0142] Based on this, the elements in the weight matrix can be distributed in an orderly manner according to their corresponding degree of influence; furthermore, elements with similar degrees of influence can be grouped into the same block, thereby achieving precise division of the weight matrix and improving the accuracy of each weight submatrix.

[0143] Step 103: Based on the channel sensitivity of the network channel corresponding to each element in each weight submatrix, allocate corresponding quantization resources to each weight submatrix.

[0144] In this step, for each weight submatrix, the channel sensitivity of each element in the weight submatrix corresponding to the network channel can be integrated to obtain the importance of the weight submatrix to each network channel. Quantization resources are allocated based on the importance of each weight submatrix. For example, weight submatrixes with higher importance are allocated more quantization resources. Quantization resources can include resources for carrying the quantized data, such as, but not limited to, the number of quantization bits and hardware storage space. In one example, a total data volume can be predefined, and different sizes of data can be allocated to different weight submatrixes, compressing each weight submatrix into a quantized submatrix no larger than the allocated data volume. The higher the importance of the weight submatrix, the larger the corresponding data volume is allocated. In another example, a specified memory space for the hardware device can be pre-configured. This hardware device can be an electronic device for deploying the quantized visual language model. After allocating the specified memory space to multiple weight submatrixes, each weight submatrix is ​​compressed into a quantized submatrix within the capacity of the allocated memory space.

[0145] like Figure 2 As shown, optimal bit allocation (OBA) is used to allocate resources to four uniformly divided blocks (b1, b2, b3, b4), each representing a weight sub-matrix. Specifically, based on the total significance of each block, a closed-form solution of Lagrange optimization is used to allocate different numbers of quantization bits to blocks with different significance levels. Blocks with higher significance are allocated more quantization bits, allowing for higher-sensitivity blocks to receive more bits. This enables quantization resources to be adaptively concentrated on important blocks with high significance and a high impact on model output, thereby improving the accuracy and precision of quantization.

[0146] Step 104: Quantize each weight submatrix based on the quantization resources corresponding to each weight submatrix to obtain multiple quantized submatrixes.

[0147] In this step, for each weight submatrix, the weight matrix is ​​quantized according to its allocated quantization resources to obtain the corresponding quantized submatrix. Based on this, multiple quantized submatrixes are obtained corresponding to multiple weight submatrixes. The more quantization resources a weight submatrix corresponds to, the more resources it consumes. The resources consumed by the quantized submatrix corresponding to a weight submatrix do not exceed the allocated quantization resources for that weight submatrix.

[0148] Specifically, during quantization, each weight submatrix corresponds to a quantization submatrix, which can be represented using a codebook and an index. Each weight submatrix can correspond to its own codebook and index. Taking Vector Quantization (VQ) as an example: For each weight submatrix, according to the quantization resources corresponding to it, the weight submatrix can be divided into multiple sub-vectors. These sub-vectors are then clustered, and a codebook for the weight submatrix is ​​created based on the cluster centers. The codeword closest to each sub-vector in the codebook is determined, and each sub-vector is mapped to the corresponding codeword's index. The quantization submatrix corresponding to the weight submatrix can include a codebook and the indices corresponding to each sub-vector. The codebook includes multiple codewords, each of which can be a vector corresponding to a cluster center, and each codeword corresponds to its own index.

[0149] Taking the number of quantization bits as an example, if the number of quantization bits corresponding to a weight submatrix is ​​larger, the codebook of the weight submatrix will be larger and the index will be more during VQ quantization.

[0150] like Figure 3As shown, taking an m×n matrix W and vector quantization as an example, the vector quantization process typically includes: reshaping the matrix W, specifically dividing the matrix W into multiple sub-vectors and recombining these sub-vectors into a matrix W' with column width v. Then, clustering is performed, specifically clustering the multiple sub-vectors into multiple clusters, and creating a v×k codebook based on the centers of these clusters as codewords. This codebook includes k codewords, each of which can be a 1×v vector. Each codeword corresponds to its own index; each sub-vector is mapped to the index of the nearest corresponding codeword in the codebook, serving as the index of that sub-vector. Based on this, the index of each sub-vector in matrix W is obtained.

[0151] For example, a weight submatrix with a larger number of quantization bits can be divided into more subvectors, resulting in more codewords in its codebook (i.e., a larger k value); or, the quantization submatrix can have more indices. Based on this, more quantization bits can be allocated to weight submatrices with higher importance, achieving finer quantization and thus improving quantization accuracy.

[0152] It should be noted that while visual language models can support the processing of multimodal data, they are prone to weight heterogeneity issues caused by multimodality. Directly using vector quantization will significantly reduce quantization accuracy. Within the same layer, the weights of a visual language model need to adapt to both image and text tokens, leading to significant heterogeneity in weight distribution. For example, ... Figure 4 As shown, this disclosure, based on research into related technologies, has found that, as Figure 4 The image tokens and text tokens output from the nine network channels shown exhibit significantly different statistical characteristics. Specifically, Figure 4 In the visual language model, there are 9 network channels, denoted as T1, T2, ..., T8, T9. Figure 4 Images (a) and (b) show the statistical characteristics of the image tokens and text tokens output by the nine network channels, respectively. Image (a) shows the similarity between image tokens output by each network channel for the same image, while image (b) shows the similarity between text tokens output by each network channel for the same text. Darker colors indicate higher similarity, with 1.0 being the highest similarity value. The comparison reveals significant differences in the characteristics of the image tokens and text tokens output by different network channels. This makes it difficult to balance the weight parameters for different characteristics during quantization.

[0153] For example, this disclosure, through research on relevant technologies, has found that the heterogeneity of weight distribution is specifically as follows: Figure 5 As shown, Figure 5 The horizontal axis, "Hidden dimension index," represents the channel index of the hidden layer in the model (i.e., the row / column dimension of the weight matrix); the vertical axis values ​​reflect the distribution characteristics of the weights of the corresponding channels. For example... Figure 5 As shown, 1 / 9999 Percentile (corresponding to the red line) indicates that after sorting the elements in the weight matrix from smallest to largest, only 1 / 9999 (approximately 0.01%) of the data are less than or equal to this value. 1 / 9999 Percentile represents the extremely low boundary of the weight distribution; the 1 / 9999 Percentile line indicates that there is significant fluctuation in the extremely low values ​​of a very small number of weights. 1 / 99 Percentile (corresponding to the purple line) indicates that after sorting the elements in the weight matrix from smallest to largest, only 1 / 99 (approximately 1%) of the data are less than or equal to this value, reflecting a low-frequency extremely low value; 1 / 99 Percentile represents the low-frequency extremely low boundary of the weight distribution; the 1 / 99 Percentile line reflects significant fluctuation in the extremely low values ​​of low-frequency weights. The 25 / 75 Percentile (corresponding to the yellow line) represents the 25th percentile (Q1) and 75th percentile (Q3), reflecting the distribution range of the middle 50% of the data; it embodies the characteristics of the middle region of the weight distribution, and the 25 / 75 Percentile line shows significant fluctuations in the concentrated distribution range of most weights. The Min / Max line (corresponding to the blue line) directly displays the minimum (Min) and maximum (Max) values ​​of the data; it characterizes the extreme range of the weight distribution, and the Min / Max line shows significant fluctuations between the minimum and maximum boundaries of the weight values. Figure 5 It can be seen that the statistical characteristics of image and text tokens are very different, and the weight distribution induced by different modalities is significantly different, resulting in a "mixed distribution" characteristic of the weight distribution.

[0154] In other words, in related technologies, if a general quantization method is adopted, a unified codebook or a unified bit configuration is difficult to take into account the weight distribution of different modes. Applying a unified codebook or fixed bit allocation across the entire layer cannot adapt to this structural heterogeneity, thereby amplifying the quantization error. That is, the quantization error is amplified in the propagation across layers and across modes.

[0155] In this embodiment, network channel sensitivity is evaluated, and the elements in the weight matrix are rearranged and partitioned based on channel sensitivity. Rearranging ensures that elements in the weight matrix are distributed in an orderly manner according to their influence on network channels, enabling precise and orderly aggregation of elements based on channel sensitivity. Partitioning groups elements with similar influence on network channels into the same weight sub-matrix, achieving refined classification of the influence of different weight parameters on the model, and providing the possibility for subsequent differentiated quantization of resources for each weight sub-matrix. Then, quantization resources are allocated based on the saliency of the weight sub-matrix; different resources are precisely allocated to different weight sub-matrixes, and the multiple quantization sub-matrixes are further corrected to obtain the target quantization matrix of the weight matrix. This effectively accommodates different weight sub-matrixes for accurate quantization, effectively overcomes the weight heterogeneity problem caused by multimodal analysis, and improves the accuracy of model quantization.

[0156] Step 105: Correct the multiple quantization sub-matrices to obtain the target quantization matrix of the weight matrix, so that the visual language model can be deployed to the hardware device to perform visual language tasks based on the target quantization matrix.

[0157] In this step, multiple quantized sub-matrices can be concatenated to obtain an initial quantization matrix. Based on the difference between the weight matrix and the initial quantization matrix, the initial quantization matrix, which introduces errors during the quantization process, is corrected to obtain a high-precision, low-bit target quantization matrix. Therefore, when deploying the quantized visual language model on hardware, the target quantization matrix can replace the weight matrix and be stored in the hardware device.

[0158] Based on this embodiment, the sensitivity of each network channel is evaluated using a weight matrix based on a visual language model, resulting in the channel sensitivity of each network channel. This weight matrix includes elements corresponding to multiple network channels. Based on the channel sensitivity of the network channels corresponding to each element in the weight matrix, the elements are rearranged and partitioned to obtain multiple weight sub-matrices. The rearrangement ensures that the elements in the weight matrix are distributed in an orderly manner according to their influence on the network channels, while the partitioning allows elements with similar influence on the network channels to cluster into the same weight sub-matrices, providing the possibility for subsequent resource differentiation quantization of each weight sub-matrice. Then, based on the channel sensitivity of the network channels corresponding to each element in each weight sub-matrice, quantization resources are allocated to each weight sub-matrice. Each weight sub-matrice is quantized based on its quantization resources to obtain multiple quantized sub-matrices. These multiple quantized sub-matrices are then corrected to obtain the target quantization matrix of the weight matrix. By rearranging and partitioning elements based on the sensitivity of the network channels corresponding to the elements, the elements in the weight matrix can be accurately and orderly aggregated according to the channel sensitivity, achieving fine classification of the influence of different weight parameters on the model. Different resources can be accurately allocated to different weight submatrices, and different weight submatrices can be compatible for accurate quantization. This can effectively overcome the weight heterogeneity problem caused by multimodality and improve the accuracy of model quantization.

[0159] Figure 6 This is a flowchart illustrating another model quantization method provided in this disclosure. In one possible embodiment, step 101 includes steps 1011-1014:

[0160] Step 1011: Based on the weight matrix and the loss function of the visual language model, evaluate the global sensitivity of the weight matrix for each network channel to obtain the global sensitivity matrix corresponding to the weight matrix.

[0161] The visual language model's network channels include input and output channels. The global sensitivity matrix comprises a first global matrix and a second global matrix. The first global matrix represents the global sensitivity of the weight matrix to each input channel, and the second global matrix represents the global sensitivity of the weight matrix to each output channel. Global sensitivity indicates the degree of influence of the weight matrix on the final output of the visual language model.

[0162] The second-order partial derivative matrix of the loss function with respect to the weight matrix characterizes the degree of influence of the weight matrix on the final output of the model. This second-order partial derivative matrix contains information about the degree of influence of both the input and output channels on the final output of the model.

[0163] As a concrete example, the second-order partial derivative matrix can be the Hessian matrix of the weight matrix; the second-order partial derivative matrix can be represented as H. The elements in H describe the local curvature of the loss function in the weight matrix space. The larger the curvature, the more drastic the change in the loss function in that direction, and the more significant the impact of weight adjustments on the model output. For example, when the curvature of the Hessian matrix is ​​large in a certain direction, it indicates that the model is very sensitive to weight changes in that direction, and the weights in that direction are more important. Subsequent quantization needs to consider their importance to avoid accuracy loss.

[0164] For example, the weight matrix W can be an m×n matrix. In the weight matrix W, m rows represent m output channels and n columns represent n input channels, where m and n are both positive integers. Then the Hessian matrix of the weight matrix W can be an mn×mn matrix.

[0165] For example, each element in the H matrix represents the joint second-order effect of two corresponding elements in the weight matrix on the loss function. Specifically, the diagonal elements in the H matrix represent the second-order effect of a single element in the weight matrix on the loss function; a larger diagonal element indicates a greater influence of that single element on the loss function. The off-diagonal elements in the H matrix represent the coupling effect between two different elements in the weight matrix. A larger off-diagonal element indicates a greater mutual influence between the adjustments of these two different elements.

[0166] For example, the input-side Hessian matrix can be decomposed from the second-order partial derivative matrix of the weight matrix as the first global matrix, and the output-side Hessian matrix can be decomposed from the second-order partial derivative matrix as the second global matrix.

[0167] Among them, the output-side Hessian matrix H O The value is m×m, representing the local curvature of the weights of the loss function with respect to the m output channels, reflecting the second-order rate of change in the output direction; H O The larger the eigenvalue of an element, the more significant the impact of the output channel weights on the loss function. The input-side Hessian matrix H... I The value is n×n, representing the local curvature of the weights of the loss function with respect to the n input channels, reflecting the second-order rate of change in the input direction; H I The larger the eigenvalue of an element, the more significant the impact of the input channel weights on the loss function. m and n are positive integers.

[0168] It should be noted that, assuming the loss function of the weight matrix is ​​E, the loss function, when expanded in second order Taylor at the current value of w, can be expressed as the following formula (1):

[0169]

[0170] In formula (1), g is the gradient of the loss function E at w, and H is the Hessian matrix of the loss function E at w.

[0171] Step 1012: Obtain the global sensitivity of each network channel based on the global sensitivity matrix.

[0172] In this step, the global sensitivity of each input channel can be obtained based on the diagonal elements of the first global matrix, and the global sensitivity of each output channel can be obtained based on the diagonal elements of the second global matrix. For example, the first global matrix H can be taken as... I The n diagonal elements are used as the global sensitivities for the n input channels. A second global matrix H can be taken. O The m diagonal elements are used as the global sensitivity of the m output channels.

[0173] Step 1013: Based on the weight matrix and the activation sample set of the visual language model, evaluate the local sensitivity of the weight matrix for each network channel to obtain the local sensitivity of each network channel.

[0174] Local sensitivity characterizes the degree of influence of the weight matrix on the output of the network layer containing the network channel; it is an assessment of the impact of the weight matrix on the local output of the network layers within the model. In this step, local sensitivity can be measured based on the contribution of a single network channel to the output activation energy.

[0175] Step 1014: Based on the global sensitivity and local sensitivity of each network channel, obtain the channel sensitivity of each network channel.

[0176] In this step, the global sensitivity and local sensitivity of each network channel can be fused to obtain the channel sensitivity. Specifically, the global sensitivity and local sensitivity can be normalized by min-max respectively; then, the global sensitivity and local sensitivity of each network channel can be fused. For example, for the i-th input channel, the global sensitivity and local sensitivity of the input channel can be fused using the following formula (2) to obtain the channel sensitivity of the i-th input channel:

[0177]

[0178] In formula (2), I (in) [i] represents the channel sensitivity of the i-th input channel. This represents the global sensitivity of the i-th input channel after normalization. This represents the local sensitivity of the i-th input channel after normalization.

[0179] Similarly, for the j-th output channel, the global sensitivity and local sensitivity of the input channel can be fused using the following formula (3) to obtain the channel sensitivity of the i-th input channel:

[0180]

[0181] In formula (3), I (out) [j] represents the channel sensitivity of the j-th output channel. This represents the global sensitivity of the j-th output channel after normalization. This represents the local sensitivity of the j-th output channel after normalization.

[0182] Based on this embodiment, by multiplying the global sensitivity and local sensitivity and then taking the logarithm, a precise fusion of global and local sensitivity can be achieved, effectively balancing the importance of network channels to both the global and local levels. This allows the channel sensitivity of a network channel to comprehensively measure its importance to both the global output of the model and the local output of the network layer, improving the accuracy of the channel sensitivity and thus enhancing the accuracy of quantization.

[0183] Figure 7 This is a flowchart illustrating another model quantization method provided in an embodiment of this disclosure. Figure 7 As shown, step 1011 can be achieved through any one of the following steps: 1011-1 or 1011-2.

[0184] Step 1011-1: Based on the gradient of the weight matrix by the loss function of the visual language model, determine the first global matrix and the second global matrix corresponding to the weight matrix.

[0185] In this step, the global sensitivity of the input and output channels can be calculated by the expectation of the gradient of the loss function with respect to the weight matrix.

[0186] For example, the first and second global matrices can be determined based on the gradient of the loss function with respect to the weight matrix using the following formula (4):

[0187]

[0188] In formula (4), H I H represents the first global matrix; O This represents the second global matrix. This represents the gradient of the loss function with respect to the weight matrix W; the loss function can be the loss function of a visual language model, which can be the KL (Kullback-Leibler, relative entropy) divergence. Here, has the same size as the weight matrix W. It is also an m×n matrix, where m and n are both positive integers.

[0189] Step 1011-2: Determine the second-order partial derivative matrix of the loss function of the visual language model with respect to the weight matrix, and decompose the second-order partial derivative matrix to obtain the first global matrix and the second global matrix corresponding to the weight matrix.

[0190] In this step, H can be decomposed into the input-side Hessian matrix H. I To capture the sensitivity of the input channel and the Hessian matrix H on the output side. O To capture the sensitivity of the output channel.

[0191] For example, the second-order partial derivative matrix H can be decomposed using the following formula (5) to obtain the first and second global matrices:

[0192]

[0193] in, H represents the Kronecker product; I This is the first global matrix, also known as the input-side Hessian matrix; H O This is the second global matrix, also known as the output-side Hessian matrix. H represents the second-order partial derivative matrix; the size of H is mn×mn.

[0194] For the Hessian matrix H of the weight matrix, we can calculate the first-order partial derivatives of the loss function with respect to each element of the weight matrix, and then calculate the second-order partial derivatives with respect to each element. Arranging the second-order partial derivatives of each element in order into a symmetric matrix yields the weight matrix H.

[0195] Based on this embodiment, the global sensitivity can be calculated using any method as needed. For example, the global sensitivity of the input and output channels can be calculated quickly and efficiently by directly using the expectation of the gradient of the loss function with respect to the weight matrix. Alternatively, the Kronecker decomposition of the second-order partial derivative matrix can be used to obtain a more accurate global sensitivity of the input and output channels. This effectively improves the practicality of the quantization method.

[0196] Figure 8 This is a flowchart illustrating another model quantization method provided in an embodiment of this disclosure. Figure 8 As shown, step 1014 can be achieved through the following steps 1014a to 1014c:

[0197] 1014a. Obtain the sample activation matrix corresponding to the activated sample set;

[0198] 1014b. Based on the elements corresponding to each input channel in the sample activation matrix and the weight matrix, determine the local sensitivity of the weight matrix for each input channel;

[0199] 1014c. Based on the elements corresponding to each output channel in the sample activation matrix and the weight matrix, determine the local sensitivity of the weight matrix to each output channel.

[0200] In this step, the activation sample set includes multiple samples, and the data type of each sample may include, but is not limited to, text, image, or audio / video data. Feature extraction can be performed on each sample in the activation sample set to obtain a sample activation matrix.

[0201] For example, the sample activation matrix can be represented as x, which can be a matrix of size k×n, where k represents the k samples included in the activation sample set; n represents the number of input feature channels for each sample, corresponding to the n input channels of the weight matrix of the visual language model. Local sensitivity can characterize the output energy generated by the sample activation matrix and W on the corresponding input or output channels.

[0202] The local sensitivity of each input channel and output channel can be obtained using the following formula (6):

[0203]

[0204] In formula (6), W represents the local sensitivity of the i-th input channel out of n input channels. ,i This represents the element in the i-th column of the m×n weight matrix W, which is also the element corresponding to the i-th input channel. This represents the transpose of the i-th column vector in the sample activation matrix x.

[0205] Among them, W: ,i The element acting on the i-th input feature channel in a k×n sample activation matrix x. That is, taking the i-th column vector W in W. :,i , including the elements of the i-th input channel; and, taking the i-th column vector in x, including the elements of the i-th input feature channel; according to W :,i transpose of the i-th column vector of x The L2 norm expectation of the product is obtained.

[0206] In formula (6), W represents the local sensitivity of the j-th output channel out of m output channels. j,: This represents the element in the j-th row of the m×n weight matrix W, which is also the element corresponding to the j-th output channel. This represents the transpose of the j-th row vector in the sample activation matrix x.

[0207] Among them, W can be j,: The element applied to the j-th sample in a k×n sample activation matrix x. That is, taking the j-th row vector W in W. j,: This includes the element corresponding to the j-th output channel; and, taking the j-th row vector in x, including the elements corresponding to each input feature channel for the j-th sample; according to W j,: transpose of the j-th row vector of x The L2 norm expectation of the product is obtained.

[0208] Based on this embodiment, the contribution of network channels to output activation energy is measured by the L2 norm expectation of the product of the input activation matrix and the corresponding weight channels, thereby accurately assessing local sensitivity and improving the accuracy of model quantization.

[0209] When quantizing a VLM model, the goal is to reduce the storage size of model parameters and the computational resource requirements while maintaining model performance as much as possible. From a single-layer perspective, the aim is to find a quantized weight. This minimizes the difference in output between the quantized weights and the unquantized weights. That is:

[0210]

[0211] In formula (7), X represents the input data of the VLM model; WX represents the output result, that is, the output result obtained after the input X is processed by the VLM model. Formula (7) is the target of quantization compensation.

[0212] The process of quantification compensation involves the second-order Taylor expansion of the loss function, which is Equation (1):

[0213]

[0214] Formula (1) is the Taylor expansion of the loss caused by quantization. T Δw represents the first-order term. This represents a second-order term.

[0215] Figure 9 This is a flowchart illustrating another model quantization method provided in this disclosure. Figure 9 As shown, in one possible embodiment, step 105 includes steps 1051-1053:

[0216] Step 1051: Obtain the initial quantization matrix of the visual language model based on the multiple quantization sub-matrices.

[0217] In this step, the initial quantization matrix can be obtained by inversely integrating multiple quantization submatrices, similar to how the weight matrix is ​​divided into multiple weight submatrices. For example, if the weight matrix is ​​evenly divided into 2×2 weight submatrices, then in this step, the initial quantization matrix is ​​obtained by concatenating the four quantization submatrices according to their corresponding weight submatric positions.

[0218] Step 1052: Perform matrix decomposition on the first global matrix and the second global matrix corresponding to the weight matrix to obtain the first lower triangular matrix and the first diagonal matrix corresponding to the first global matrix, and the second lower triangular matrix and the second diagonal matrix corresponding to the second global matrix.

[0219] In this step, the first global matrix and the second global matrix can be decomposed using Block-LDL. Specifically, based on the first global matrix and the second global matrix, Block-LDL decomposition can be performed using the following formula (8) to obtain the first lower triangular matrix, the first diagonal matrix, the second lower triangular matrix, and the second diagonal matrix:

[0220]

[0221] In formula (8), L I H represents the first global matrix. I The first lower triangular matrix is ​​of size n×n. I The diagonal elements are 1s, and the elements above the diagonal elements are 0s. L I Records the Hessian matrix H of the input channels after elimination I The entire process of "variable cross-coupling terms" involves each column corresponding to a "coupling elimination operation" for a variable, to avoid mutual interference between weight parameters during subsequent compensation. I Except for the diagonal elements, all other elements are cross-coupled variable terms.

[0222] L O H represents the second global matrix O The second lower triangular matrix has a size of m×m, L O The diagonal elements are all 1s, and the elements above the diagonal elements are all 0s. This is a unit lower triangular matrix, L. O The output channel Hessian matrix H is recorded. O The entire process of "variable cross-coupling terms" involves each column corresponding to a "coupling elimination operation" for a variable, to avoid mutual interference between weight parameters during subsequent compensation. O Except for the diagonal elements, all other elements are cross-coupled variable terms.

[0223] D I H represents the first global matrix. IThe first diagonal matrix is ​​the principal curvature value of each input channel after decoupling, and its size is n×n.

[0224] D O H represents the second global matrix O The second diagonal matrix is ​​the principal curvature value of each output channel after decoupling, and its size is m×m. I is the identity matrix.

[0225] It should be noted that after Block-LDL decomposition, the matrix can be obtained as a lower triangular matrix and a diagonal matrix with all diagonal elements being positive.

[0226] Step 1053: Based on the first lower triangular matrix, the second lower triangular matrix, the first diagonal matrix, and the second diagonal matrix, the initial quantization matrix is ​​corrected to obtain the target quantization matrix.

[0227] In this step, based on the first lower triangular matrix, the second lower triangular matrix, the first diagonal matrix, and the second diagonal matrix, the first-order compensation term and the second-order compensation term can be determined, and the initial quantization matrix can be corrected based on the first-order compensation term and the second-order compensation term. The first-order compensation term can be the compensation term for the first-order gradient of the loss function with respect to the weight matrix. The second-order compensation term can be the compensation term for the second-order gradient of the loss function with respect to the weight matrix.

[0228] Based on this embodiment, the input and output Hessian matrices are used to compensate for errors in the initial quantization matrix. This approach takes into account the curvature of the loss function's effect from parameter variations, thereby allowing for more precise adjustment of the quantized weights and improving the accuracy of quantization compensation.

[0229] Figure 10 This is a flowchart illustrating another model quantization method provided in this disclosure. In one possible embodiment, an iterative approach can be used to repeatedly correct the initial quantization matrix, so that the iterative quantization matrix obtained during the iteration process continuously approximates the weight matrix. In one possible example, such as... Figure 10 As shown, step 1053 includes:

[0230] Iteratively execute the following steps S1 to S7 until the preset convergence condition is met, then stop the iteration and use the iterative quantization matrix obtained in the last iteration as the target quantization matrix:

[0231] The convergence condition can be set according to actual needs. For example, it may include: the difference between the iterative quantization matrix obtained in this iteration and the quantization matrix to be corrected in this iteration is not higher than a preset difference threshold, or the number of times the following steps S1 to S7 are executed reaches a preset number (e.g., 1000 times), etc. The embodiments of this disclosure do not limit the convergence condition.

[0232] S1. Use the iterative quantization matrix obtained in the previous iteration as the quantization matrix to be corrected in this iteration, and the quantization matrix to be corrected in the first iteration is the initial quantization matrix;

[0233] The quantization matrix to be corrected is the iterative quantization matrix that needs to be corrected in this iteration. For the first iteration, the quantization matrix to be corrected is the initial quantization matrix. For the second and subsequent iterations, the quantization matrix to be corrected is the iterative quantization matrix obtained in the previous iteration.

[0234] S2. Determine the residual between the weight matrix and the quantization matrix to be corrected;

[0235] The residual directly reflects the quantization error corresponding to the quantization matrix to be corrected. In this step, the residual between the original weight matrix and the quantization matrix to be corrected in this iteration is calculated using the following formula (9):

[0236]

[0237] In formula (9), E represents the residual. This represents the quantization matrix to be corrected. For the first iteration, This is the initial quantization matrix.

[0238] During multiple iterations, the residual can be used as the core optimization variable. Specifically, each iteration can further optimize the residual from the previous iteration, so that the iterative quantization matrix obtained in each iteration can gradually approximate the original weight matrix, ultimately obtaining the target quantization matrix for high-precision compression at low bit rates.

[0239] S3. Based on the residual, the first lower triangular matrix, and the second lower triangular matrix, determine the first-order compensation term;

[0240] In this step, based on the residual, the first lower triangular matrix, and the second lower triangular matrix, we can obtain... and "E·L" I "These two terms serve as first-order compensation terms. The first-order compensation term represents the loss caused by quantization in the quantization weights to be corrected." The first-order term of the Taylor expansion at that point.

[0241] S4. Perform inverse transformation on the first lower triangular matrix and the second lower triangular matrix to obtain the intermediate matrix. Determine the correction coefficients based on the first diagonal matrix and the second diagonal matrix, and obtain the interactive compensation term based on the intermediate matrix and the correction coefficients.

[0242] In this step, the first and second lower triangular matrices can be inversely transformed using the following formula (10) to obtain the intermediate matrix:

[0243] A = (L O +I)-T ·E·(L I +I) -1 ; Formula (10)

[0244] Where A represents the intermediate matrix, with a size of m×n. A is the input-output cross-compensation matrix, representing the first-order term error of the Taylor expansion of the quantization loss applied to the quantization weights to be corrected. The response value of the interaction between the input and output channels. E represents the residual.

[0245] Furthermore, the correction factor can be determined using the following formula (11):

[0246]

[0247] Among them, Γ ij This represents the element in the i-th row and j-th column of the correction coefficient Γ. D O (i) represents the diagonal element corresponding to the i-th row in the second diagonal matrix, D I (j) represents the diagonal element corresponding to the j-th column in the first diagonal matrix.

[0248] Γ represents the importance of the cross-compensation matrix A, which is a coefficient matrix that needs to be determined by adjusting the parameter β. Specifically, the correction coefficient Γ of size m×n can be obtained by element-wise calculation using formula (11).

[0249] Where β is the expression that satisfies D O (i)·D I (j)>2β is a constant to ensure the positive definiteness of the coefficients.

[0250] Based on this, the error weights of high gradient layers can be reduced by adjusting the coefficients, thus avoiding exponential amplification of errors.

[0251] Furthermore, based on the intermediate matrix and the correction coefficients, the interaction compensation term is obtained using the following formula (12):

[0252]

[0253] Where T represents the interactive compensation term, This indicates a dot product. T represents the cross-effect between the first and second-order terms of the Taylor expansion of the quantization loss.

[0254] S5. Based on the residual, the first lower triangular matrix, and the second lower triangular matrix, determine the second-order compensation term;

[0255] In this step, based on the residual, the first lower triangular matrix, and the second lower triangular matrix, we can obtain... As a second-order compensation term. The second-order compensation term represents the loss caused by quantization in the quantization weights to be corrected. The second-order term of the Taylor expansion at the location.

[0256] S6. Based on the first-order compensation term, the interaction compensation term, and the second-order compensation term, the weight matrix is ​​corrected to obtain the correction matrix for this iteration.

[0257] In this step, the weight matrix can be corrected using the following formula (13) to obtain the corrected matrix for this iteration:

[0258]

[0259] Where η represents the correction matrix. η is obtained by compensating for first-order and second-order terms of W. The purpose is to make the VLM model output based on the quantization matrix obtained by quantizing η according to steps 101 to 104 approximate the VLM model output based on the original weight matrix W by iteratively executing S6, thereby continuously weakening the loss caused by quantization.

[0260] S7. Quantize the correction matrix for this iteration to obtain the iteration quantization matrix for this iteration;

[0261] In this step, the same process as in steps 101 to 104 can be used to quantize the correction matrix, obtain multiple quantized correction sub-matrices corresponding to the correction matrix, and integrate the multiple quantized correction sub-matrices to obtain the iterative quantization matrix.

[0262] In one possible example, the correction matrix η can be input into the quantizer Q to obtain the iterative quantization matrix for this iteration. The quantizer Q integrates a quantization process similar to steps 101 to 104.

[0263] Then, based on the iterative quantization matrix, it can be determined whether the current iteration has reached the convergence condition. Specifically, it can be calculated using the following formula (14). The quantization matrix to be corrected in this iteration Relative differences in F-norm:

[0264]

[0265] Where Δ represents and The relative differences in the F-norm.

[0266] If Δ < ε, it indicates that the convergence condition has been met, and ε represents the preset difference threshold. If the convergence condition has not been met, S1-S7 are executed again.

[0267] It should be noted that, in this embodiment, a sensitivity-driven structured mixed-precision quantization (SSMQ) module can be designed to implement the process described in steps 101 to 104 above to obtain multiple quantization sub-matrices, and the process of integrating the multiple quantization sub-matrices to obtain an initial quantization matrix. Furthermore, a gradient-aware error compensation (GAEC) module can also be designed to implement this: for each layer, by analyzing the Taylor expansion of the global loss, the first-order gradient matrix is ​​approximated using the quantization residuals, and the second-order partial derivative matrix is ​​approximated using Kronecker decomposition; based on this, multiple iterations are performed to continuously approximate the weight matrix with the iterative quantization matrix, thereby gradually reducing the quantization error.

[0268] Figure 11 This is a statistical diagram of the first-order gradient. For example... Figure 11 As shown, the horizontal axis lists the various network modules of the model: query module (q_proj), key module (k_proj), value module (v_proj), output module (o_proj), upsampling module (up_proj), gate module (gate_proj), and downsampling module (down_proj). The vertical axis ranges from 0 to 0.025, representing the 99th percentile of the first-order gradient of the corresponding module. The quantile describes the tail characteristics of the data distribution; the 99th percentile of the first-order gradient indicates that 99% of the gradient values ​​in this module are smaller than the value corresponding to this quantile, reflecting the distribution level of large gradient values.

[0269] like Figure 11 As shown, the 99th percentile of the first-order gradient differs significantly among different modules. The quantile of the key module is close to 0.022, the largest among all modules; while the quantiles of the query and gating modules are only close to 0.002, which is relatively small; the quantiles of the remaining modules fluctuate around 0.005. Clearly, these quantile values ​​are not close to 0, indicating that the first-order gradient has a substantial and non-negligible contribution to the loss of model parameters. In other words, the first-order gradient plays a dominant role in the residual of the model loss (loss fluctuation caused by quantization error). If quantization compensation relies solely on the second derivative and ignores the contribution of the first-order gradient, the compensation result will deviate from the actual requirements (bias), ultimately leading to a decline in model performance.

[0270] Based on this embodiment, by iteratively executing steps S1 to S7 in step 1053, the initial quantization matrix is ​​further iteratively corrected by combining the first-order compensation term with the second-order compensation term. This compensates for the dominant contribution of the first-order gradient to the residual, suppresses the accumulation of cross-layer errors, and enables the iterative quantization matrix obtained in each iteration to effectively approximate the weight matrix. This achieves high compression ratio, low bandwidth usage, and near-floating-point accuracy at extremely low bit depths, thereby improving the accuracy of quantization.

[0271] Figure 12 This is a flowchart illustrating another model quantization method provided in this disclosure. Figure 12 As shown, in one possible embodiment, step 102 includes steps 1021-1022:

[0272] Step 1021: Based on the channel sensitivity of the output channel corresponding to each element in the weight matrix, sort each row in the weight matrix in descending order of channel sensitivity; and based on the channel sensitivity of the input channel corresponding to each element in the weight matrix, sort each column in the weight matrix in descending order of channel sensitivity, to obtain the rearranged weight matrix.

[0273] Step 1022: Divide the rearranged weight matrix into multiple blocks to obtain multiple weight submatrices.

[0274] In this step, for a weight matrix W of size m×n, we first sort the m rows of W in descending order according to the channel sensitivity of the m output channels. After sorting, the channel sensitivity of the output channel corresponding to the element in the (i-1)th row of W is greater than the channel sensitivity of the output channel corresponding to the element in the ith row. Then, we sort the n columns of W in descending order according to the channel sensitivity of the n input channels. After sorting, the channel sensitivity of the input channel corresponding to the element in the (j-1)th column of W is greater than the channel sensitivity of the input channel corresponding to the element in the jth column.

[0275] Furthermore, segmentation points can be determined along the input and output dimensions of the weight matrix to divide the rearranged weight matrix into multiple structured blocks, each block being a weight submatrix. For example, the weight matrix can be uniformly divided into k×k blocks; for example, determining one segmentation point each along the input and output dimensions yields 2×2 blocks, or four blocks; alternatively, it can be divided into 3×3, 5×5 blocks, etc. Of course, uniform division is not always necessary; for example, it can be divided into two blocks along the diagonal lines in the lower left and upper right directions. This embodiment does not specifically limit the division method or the number of blocks after division.

[0276] Based on this embodiment, by rearranging the elements according to the sensitivity of the network channel, the elements in the rearranged weight matrix are distributed in an orderly manner according to their influence on the network channel. Through partitioning, the influence of each element in each weight submatrix on the network channel is made similar; for example, elements with a greater influence on the network channel are assigned to one weight submatrix, and elements with a smaller influence on the network channel are assigned to another weight submatrix. This facilitates subsequent quantization of different weight submatrixes using different quantization resources, improving the precision and accuracy of weight matrix quantization.

[0277] Figure 13 This is a flowchart illustrating another model quantization method provided in this disclosure. Figure 13 As shown, in one possible embodiment, step 103 includes the following steps 1031-1033:

[0278] Step 1031: For each weight submatrix, determine the significance of each element in each weight submatrix based on the channel sensitivity of the input channel and the channel sensitivity of the output channel corresponding to each element in each weight submatrix.

[0279] In this step, the salience of an element can characterize the overall influence of that element on its corresponding input and output channels. In this embodiment of the disclosure, the higher the overall influence of an element, the higher the salience of that element relative to other elements.

[0280] For example, the significance of each element in the weighted submatrix can be defined as "the sensitivity of the corresponding output channel × the sensitivity of the input channel", that is, the significance of each element can be obtained through the following formula (15):

[0281] I i,j =I (out) [i]·I (in) [j]; Formula (15)

[0282] In formula (15), I i,j Let I represent the significance of the element in the i-th row and j-th column of W; (out) [i] represents the channel sensitivity of the element corresponding to the i-th output channel out of m output channels, I (in) [j] indicates the channel sensitivity of the element corresponding to the j-th input channel out of n input channels.

[0283] Step 1032: Determine the significance of each weight submatrix based on the significance of multiple elements in each weight submatrix.

[0284] In this step, for each weighted submatrix, the sum of the saliencies of its elements can be used as the saliency of that submatrix. The saliency of a weighted submatrix represents its relative importance among all weighted submatrixes. The higher the saliency of a weighted submatrix, the higher its relative importance.

[0285] Step 1033: Based on the significance of each weight submatrix, allocate corresponding quantization resources to each weight submatrix.

[0286] In this step, the higher the significance of a weighted submatrix, the higher its relative importance compared to other weighted submatrixes, and the more quantization resources it is allocated to.

[0287] In one possible example, the quantization resource may include the number of quantization bits, which can represent the spatial size of the quantized data corresponding to the weight submatrix. For instance, the number of quantization bits corresponding to each weight submatrix can be determined based on the saliency of each weight submatrix using the following formula (16):

[0288]

[0289] In formula (16), b t S represents the number of quantization bits corresponding to the t-th weight submatrix; B is the total budget bits, which can be predetermined based on hardware resources; S t S represents the significance of the t-th weight submatrix. s This represents the significance of the s-th weight submatrix among multiple weight submatrixes. In formula (16), only the division of the weight matrix into 4 weight submatrixes is shown as an example. The number of weight submatrix divisions in this embodiment is not limited to this. Of course, the weight matrix can also be divided into 9, 16, 25 or other different numbers of blocks. The number of weight submatrixes is not limited in this embodiment.

[0290] like Figure 2 As shown, after reordering by MRSBP, the elements are arranged in order of their corresponding significance, and the elements with high significance are clustered in the left and upper regions of the weight matrix.

[0291] Based on this embodiment, by allocating quantization resources according to the saliency of the weighted submatrix, priority quantization resources are given to weighted submatrixes with relatively high importance, while weighted submatrixes with relatively low importance receive relatively fewer quantization resources. This achieves a fine allocation of "more bits in high-sensitivity regions and fewer bits in low-sensitivity regions," allowing limited quantization resources to be precisely concentrated in the relatively more important regions of the weighted matrix. This enables high compression ratios and low bandwidth usage at extremely low bit depths, thereby improving the accuracy of quantization.

[0292] Figure 14 This is a flowchart illustrating a model inference method provided in this disclosure. This method can be applied to electronic devices. After quantization based on steps 101-105, the quantized VLM model can be deployed to hardware devices. This disclosure is applicable to the deployment of large-scale VLM models in resource-constrained environments, achieving high compression ratios, low bandwidth usage, and near-floating-point precision with extremely low bit depth. It is particularly suitable for edge devices with limited hardware computing resources, GPU-constrained devices, and devices providing high-concurrency inference services. Specifically, it can be used on computing platforms of electronic devices, which may include AI computing units capable of calling VLM models to process various computing tasks. Edge devices include various computing devices deployed on edge nodes, including smart vehicles, robots, smartphones, smart homes, cameras, etc. Edge devices are closer to the data source and users, and have higher real-time requirements; quantization enables the model to meet these requirements.

[0293] like Figure 14 As shown, the model inference method may include the following steps 201:

[0294] Step 201: In response to receiving a visual language task, call the visual language model to process the natural language instructions and visual information in the visual language task and obtain the task processing result;

[0295] The target quantization matrix obtained by the aforementioned model quantization method is pre-deployed to the hardware device. In one possible example, the quantized VLM model performs a visual language task based on the target quantization matrix. When a visual language task is received, the target quantization matrix of the visual language model is retrieved from the hardware device's storage space. Based on this target quantization matrix, the natural language instructions and visual information are processed to obtain the task processing result. Furthermore, this output result can be returned to the user.

[0296] The natural language instruction can be an operation command issued in natural language form to trigger the execution of a visual language task. For example, natural language instructions may include, but are not limited to: "Based on the image content, answer the following questions:...", "Generate a detailed description for this product image", "Describe the traffic scene shown in these images", "Generate a painting based on the following description:...", "Which city is the geographic coordinate in the image located in?", etc. For example, visual information may include, but is not limited to: images, videos, or multimodal data including images, as well as related text, audio and video data, and sensor data.

[0297] After the quantized VLM model is deployed on the hardware device, it can support the processing of various visual language tasks. The following are examples of possible task scenarios for the quantized VLM model.

[0298] In one scenario example, a hardware device can invoke a quantized VLM model to perform an image description generation task. For instance, the VLM model automatically generates a natural language description for an image input by the user.

[0299] In another scenario example, the hardware device can invoke a quantized VLM model to perform visual question answering tasks. For instance, the VLM model can answer questions from users about the content of an image based on that image. Furthermore, users can also engage in two-way dialogue with the VLM model and add images to the context of the conversation.

[0300] In another scenario example, the hardware device can invoke a quantized VLM model to perform cross-modal retrieval tasks. For instance, the VLM model retrieves relevant images based on text descriptions.

[0301] In another scenario example, hardware devices can invoke a quantized VLM model to perform autonomous driving tasks. For instance, the VLM model processes natural language commands input by the user and images captured by the vehicle's cameras, outputting driving actions, warning prompts, etc., to help the vehicle understand complex road environments and traffic rules.

[0302] In addition, the VLM model can be integrated into visual agents to autonomously perform vision-related tasks.

[0303] In another scenario example, VLM models can be integrated into larger systems to build video analytics AI agents capable of detecting specific events when prompted. These systems could be used to detect malfunctioning robots in a warehouse or to issue out-of-stock alerts when shelves become empty.

[0304] In another scenario example, VLM models can be used in conjunction with technologies such as graph databases to understand long videos, helping them capture complex objects and activities within the video. Such systems can be used to summarize operations in a warehouse to identify bottlenecks and inefficiencies, or to create sports commentary for football, basketball, or other sporting events.

[0305] Based on the embodiments of this disclosure, when a visual language task is received, a visual language model is invoked to process the natural language instructions and visual information in the visual language task, thereby obtaining the task processing result. Since the visual language model is based on a target quantization matrix obtained using the aforementioned model quantization method, which is pre-deployed on the hardware device, this target quantization matrix is ​​accurately quantized to be compatible with different weight submatrices, overcoming the weight heterogeneity problem caused by multimodal processing. Therefore, performing the visual language task based on this target quantization matrix yields the output result, improving the accuracy of model inference.

[0306] It is understood that the VLM model in this disclosure is a multimodal, generative AI model capable of simultaneously understanding and processing video, image, and text information. This type of model enables the AI ​​system to "see" by combining a Large Language Model (LLM) with a visual encoder. With this capability, the VLM model can process and provide a high-level understanding of video, image, and text input in prompts to generate text responses. For example, the VLM model supports open-ended user input prompts, allowing users to instruct the VLM model to answer questions, summarize, explain content, or use images for reasoning. Users can also engage in bidirectional dialogue with the VLM model and add images to the context of the dialogue.

[0307] Large language models refer to deep learning models trained on massive amounts of text data. These models can generate natural language text or understand spoken language text, and can process various types of text content from input text data, including conversational AI, chatbots, marketing content, and code assistants. Multimodal models can process input text, images, videos, audio, and other multimodal information. The inputs and outputs of multimodal models can include multiple forms, such as text, images, audio, and video.

[0308] As a possible example, the VLM model architecture consists of three parts: a visual encoder, a projector, and an LLM. The visual encoder is typically a CLIP (Contrastive Language-Image Pre-Training) model based on a transformer architecture. This CLIP model has been trained on millions of image-text pairs and possesses the ability to associate images with text. The projector, composed of a set of network layers, transforms the output of the visual encoder into a format that the LLM can understand, generally interpreting it as image tokens. Existing LLMs can be used to construct VLMs, and there are currently several VLM model variants that combine various LLMs with visual encoders.

[0309] Any of the model quantization methods and model inference methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, any of the model quantization methods and model inference methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the model quantization methods and model inference methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0310] Figure 15 This is a schematic diagram of the structure of a model quantization device provided in an embodiment of this disclosure. Figure 15 As shown, the device includes:

[0311] Evaluation module 301 is used to evaluate the sensitivity of each network channel of the visual language model based on the weight matrix of the visual language model, and obtain the channel sensitivity of each network channel. The weight matrix includes elements corresponding to multiple network channels.

[0312] The partitioning module 302 is used to rearrange and partition the elements in the weight matrix based on the channel sensitivity of the network channels corresponding to each element in the weight matrix, so as to obtain multiple weight submatrices.

[0313] The allocation module 303 is used to allocate corresponding quantization resources to each weight submatrix based on the channel sensitivity of the network channel corresponding to each element in each weight submatrix.

[0314] The determination module 304 is used to perform quantization processing on each weight submatrix based on the quantization resources corresponding to each weight submatrix, so as to obtain multiple quantized submatrixes.

[0315] The correction module 305 is used to correct the multiple quantization sub-matrices to obtain the target quantization matrix of the weight matrix, so that the visual language model can be deployed to the hardware device to perform visual language tasks based on the target quantization matrix.

[0316] In one possible implementation, the evaluation module 301 includes:

[0317] The first evaluation unit is used to evaluate the global sensitivity of the weight matrix for each network channel based on the weight matrix and the loss function of the visual language model, and obtain the global sensitivity matrix corresponding to the weight matrix.

[0318] The first determining unit is used to obtain the global sensitivity of each network channel based on the global sensitivity matrix;

[0319] The second evaluation unit is used to evaluate the local sensitivity of the weight matrix for each network channel based on the weight matrix and the activation sample set of the visual language model, and obtain the local sensitivity of each network channel.

[0320] The second determining unit is used to obtain the channel sensitivity of each network channel based on the global sensitivity and local sensitivity of each network channel.

[0321] In one possible implementation, each network channel includes an input channel and an output channel; the global sensitivity matrix includes a first global matrix and a second global matrix, the first global matrix representing the global sensitivity of the weight matrix to each input channel, and the second global matrix representing the global sensitivity of the weight matrix to each output channel.

[0322] This first evaluation unit is used for any of the following:

[0323] Based on the gradient of the weight matrix by the loss function of the visual language model, the first global matrix and the second global matrix corresponding to the weight matrix are determined.

[0324] Determine the second-order partial derivative matrix of the loss function of the visual language model with respect to the weight matrix, and decompose the second-order partial derivative matrix to obtain the first global matrix and the second global matrix corresponding to the weight matrix.

[0325] Accordingly, the first determining unit is used to obtain the global sensitivity of each input channel based on the diagonal elements in the first global matrix, and to obtain the global sensitivity of each output channel based on the diagonal elements in the second global matrix.

[0326] In one possible implementation, the second evaluation unit is used for:

[0327] Obtain the sample activation matrix corresponding to the activated sample set;

[0328] Based on the elements corresponding to each input channel in the sample activation matrix and the weight matrix, the local sensitivity of the weight matrix to each input channel is determined.

[0329] Based on the elements corresponding to each output channel in the sample activation matrix and the weight matrix, the local sensitivity of the weight matrix to each output channel is determined.

[0330] In one possible implementation, the correction module 305 includes:

[0331] The third determining unit is used to obtain the initial quantization matrix of the visual language model based on the multiple quantization sub-matrices;

[0332] The decomposition unit is used to perform matrix decomposition on the first global matrix and the second global matrix corresponding to the weight matrix respectively, to obtain the first lower triangular matrix and the first diagonal matrix corresponding to the first global matrix, and the second lower triangular matrix and the second diagonal matrix corresponding to the second global matrix.

[0333] The correction unit is used to correct the initial quantization matrix based on the first lower triangular matrix, the second lower triangular matrix, the first diagonal matrix, and the second diagonal matrix to obtain the target quantization matrix.

[0334] In one possible implementation, when the correction unit corrects the initial quantization matrix based on the first lower triangular matrix, the second lower triangular matrix, the first diagonal matrix, and the second diagonal matrix to obtain the target quantization matrix, it is specifically used for:

[0335] The following steps are performed iteratively until the convergence condition is met, at which point the iteration stops, and the iterative quantization matrix obtained in the last iteration is used as the target quantization matrix:

[0336] The iterative quantization matrix obtained in the previous iteration is used as the quantization matrix to be corrected in the current iteration, and the quantization matrix to be corrected in the first iteration is used as the initial quantization matrix.

[0337] Determine the residual between the weight matrix and the quantization matrix to be corrected;

[0338] Based on the residual, the first lower triangular matrix, and the second lower triangular matrix, the first-order compensation term is determined.

[0339] The first and second lower triangular matrices are inversely transformed to obtain the intermediate matrix. The correction coefficients are determined based on the first and second diagonal matrices, and the interaction compensation term is obtained based on the intermediate matrix and the correction coefficients.

[0340] Based on the residual, the first lower triangular matrix, and the second lower triangular matrix, the second-order compensation term is determined;

[0341] The weight matrix is ​​corrected based on the first-order compensation term, the interaction compensation term, and the second-order compensation term to obtain the correction matrix for this iteration.

[0342] The correction matrix for this iteration is quantized to obtain the iteration quantization matrix for this iteration;

[0343] The convergence condition includes that the difference between the iterative quantization matrix obtained in this iteration and the quantization matrix to be corrected in this iteration is not higher than a preset difference threshold.

[0344] In one possible implementation, the partitioning module 302 is used for:

[0345] Based on the channel sensitivity of the output channel corresponding to each element in the weight matrix, the rows of the weight matrix are sorted in descending order of channel sensitivity. Also, based on the channel sensitivity of the input channel corresponding to each element in the weight matrix, the columns of the weight matrix are sorted in descending order of channel sensitivity, resulting in a rearranged weight matrix.

[0346] The rearranged weight matrix is ​​divided into multiple blocks to obtain multiple weight submatrices.

[0347] In one possible implementation, the allocation module 303 is used for:

[0348] For each weight submatrix, the significance of each element in each weight submatrix is ​​determined based on the channel sensitivity of the input channel and the channel sensitivity of the output channel corresponding to each element in each weight submatrix.

[0349] The significance of each weight submatrix is ​​determined based on the significance of multiple elements in each weight submatrix;

[0350] Based on the significance of each weight submatrix, corresponding quantization resources are allocated to each weight submatrix.

[0351] The model quantization device disclosed herein evaluates the channel sensitivity of each network channel using a weight matrix based on a visual language model. This weight matrix includes elements corresponding to multiple network channels. Based on the channel sensitivity of the network channels corresponding to each element in the weight matrix, the elements in the weight matrix are rearranged and divided to obtain multiple weight sub-matrices. This ensures that the elements in the weight matrix are distributed in an ordered manner according to their degree of influence, with elements of similar influence grouped into the same weight sub-matrices, providing the possibility for subsequent differentiated quantization of resources for different weight sub-matrices. Then, based on the channel sensitivity of the network channels corresponding to each element in each weight sub-matrice, quantization resources are allocated to each weight sub-matrice; each weight sub-matrice is quantized based on the quantization resources to obtain multiple quantized sub-matrices; these multiple quantized sub-matrices are then corrected to obtain the target quantization matrix. Because the elements in the weight matrix are rearranged and partitioned, they are precisely and orderly clustered according to channel sensitivity, thereby finely classifying the degree of influence of different weight parameters on the model; and different resources are precisely allocated to different weight submatrices, and different weight submatrices are compatible for accurate quantization, so that quantization resources are concentrated in more important areas of the weight matrix, which can effectively overcome the weight heterogeneity problem caused by multimodality and improve the accuracy of model quantization.

[0352] It should be noted that the specific implementation of the model quantization device in this disclosure is similar to the specific implementation of the model quantization method in this disclosure, and the technical effects of the model quantization device in this disclosure are similar to the technical effects of the model quantization method in this disclosure. For details, please refer to the description of the model quantization method section. In order to reduce redundancy, it will not be repeated.

[0353] Figure 16 This is a schematic diagram of the structure of a model inference device provided in an embodiment of this disclosure. Figure 16 As shown, the device includes:

[0354] Module 401 is used to respond to a received visual language task by calling a visual language model to process the natural language instructions and visual information in the visual language task and obtain the task processing result.

[0355] The target quantization matrix obtained by the visual language model based on the above model quantization method is pre-deployed to the hardware device.

[0356] It should be noted that the specific implementation of the model inference device in this embodiment is similar to the specific implementation of the model inference method in this embodiment, and the technical effects of the model inference device in this embodiment are similar to the technical effects of the model inference method in this embodiment. For details, please refer to the description of the model inference method section. In order to reduce redundancy, it will not be repeated.

[0357] Below, for reference Figure 17 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0358] Figure 17 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0359] like Figure 17 As shown, the electronic device includes one or more processors and memory.

[0360] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0361] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the model quantization methods, model inference methods, and / or other desired functions of the various embodiments of this disclosure described above.

[0362] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0363] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0364] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0365] Of course, for the sake of simplicity, Figure 17 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0366] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the model quantization method and model inference method according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0367] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0368] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the model quantization method and model inference method according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0369] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0370] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0371] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0372] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0373] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0374] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0375] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0376] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A model quantization method, characterized in that, include: Based on the weight matrix of the visual language model, the sensitivity of each network channel of the visual language model is evaluated to obtain the channel sensitivity of each network channel. The weight matrix includes elements corresponding to multiple network channels. Based on the channel sensitivity of the network channels corresponding to each element in the weight matrix, the elements in the weight matrix are rearranged and divided to obtain multiple weight submatrices. Based on the channel sensitivity of the network channel corresponding to each element in each weight submatrix, corresponding quantization resources are allocated to each weight submatrix. Based on the quantization resources corresponding to each weight submatrix, each weight submatrix is ​​quantized to obtain multiple quantized submatrixes. The plurality of quantization sub-matrices are modified to obtain the target quantization matrix of the weight matrix, so that the visual language model can be deployed to a hardware device to perform visual language tasks based on the target quantization matrix.

2. The method according to claim 1, characterized in that, The weight matrix based on the visual language model is used to evaluate the sensitivity of each network channel of the visual language model, obtaining the channel sensitivity of each network channel, including: Based on the weight matrix and the loss function of the visual language model, the global sensitivity of the weight matrix for each network channel is evaluated to obtain the global sensitivity matrix corresponding to the weight matrix. The global sensitivity of each network channel is obtained based on the global sensitivity matrix. Based on the weight matrix and the activation sample set of the visual language model, the local sensitivity of the weight matrix for each network channel is evaluated to obtain the local sensitivity of each network channel. The channel sensitivity of each network channel is obtained based on the global sensitivity and local sensitivity of each network channel.

3. The method according to claim 2, characterized in that, The network channels include input channels and output channels; the global sensitivity matrix includes a first global matrix and a second global matrix, wherein the first global matrix represents the global sensitivity of the weight matrix to each input channel, and the second global matrix represents the global sensitivity of the weight matrix to each output channel; The loss function based on the weight matrix and the visual language model is used to evaluate the global sensitivity of the weight matrix for each network channel, resulting in a global sensitivity matrix corresponding to the weight matrix, including any one of the following: Based on the gradient of the loss function of the visual language model with respect to the weight matrix, determine the first global matrix and the second global matrix corresponding to the weight matrix; Determine the second-order partial derivative matrix of the loss function of the visual language model with respect to the weight matrix, and decompose the second-order partial derivative matrix to obtain the first global matrix and the second global matrix corresponding to the weight matrix; The process of obtaining the global sensitivity of each network channel based on the global sensitivity matrix includes: The global sensitivity of each input channel is obtained based on the diagonal elements of the first global matrix, and the global sensitivity of each output channel is obtained based on the diagonal elements of the second global matrix.

4. The method according to claim 2, characterized in that, The local sensitivity of the weight matrix for each network channel is evaluated based on the activation sample set of the weight matrix and the visual language model to obtain the local sensitivity of each network channel, including: Obtain the sample activation matrix corresponding to the activated sample set; Based on the elements corresponding to each input channel in the sample activation matrix and the weight matrix, the local sensitivity of the weight matrix to each input channel is determined; Based on the elements corresponding to each output channel in the sample activation matrix and the weight matrix, the local sensitivity of the weight matrix to each output channel is determined.

5. The method according to claim 1, characterized in that, The step of modifying the plurality of quantization sub-matrices to obtain the target quantization matrix of the weight matrix includes: The initial quantization matrix of the visual language model is obtained based on the multiple quantization sub-matrices; Matrix decomposition is performed on the first global matrix and the second global matrix corresponding to the weight matrix to obtain the first lower triangular matrix and the first diagonal matrix corresponding to the first global matrix, and the second lower triangular matrix and the second diagonal matrix corresponding to the second global matrix. Based on the first lower triangular matrix, the second lower triangular matrix, the first diagonal matrix, and the second diagonal matrix, the initial quantization matrix is ​​modified to obtain the target quantization matrix.

6. The method according to claim 5, characterized in that, The step of modifying the initial quantization matrix based on the first lower triangular matrix, the second lower triangular matrix, the first diagonal matrix, and the second diagonal matrix to obtain the target quantization matrix includes: The following steps are performed iteratively until the convergence condition is met, at which point the iteration stops, and the iterative quantization matrix obtained in the last iteration is used as the target quantization matrix: The iterative quantization matrix obtained in the previous iteration is used as the quantization matrix to be corrected in the current iteration, and the quantization matrix to be corrected in the first iteration is used as the initial quantization matrix. Determine the residual between the weight matrix and the quantization matrix to be corrected; Based on the residual, the first lower triangular matrix, and the second lower triangular matrix, the first-order compensation term is determined; The first lower triangular matrix and the second lower triangular matrix are inversely transformed to obtain the intermediate matrix. The correction coefficients are determined based on the first diagonal matrix and the second diagonal matrix, and the interaction compensation term is obtained based on the intermediate matrix and the correction coefficients. Based on the residual, the first lower triangular matrix, and the second lower triangular matrix, the second-order compensation term is determined; The weight matrix is ​​modified based on the first-order compensation term, the interaction compensation term, and the second-order compensation term to obtain the modified matrix for this iteration. The correction matrix of this iteration is quantized to obtain the iteration quantization matrix of this iteration; The convergence condition includes that the difference between the iterative quantization matrix obtained in this iteration and the quantization matrix to be corrected in this iteration is not higher than a preset difference threshold.

7. The method according to any one of claims 1-6, characterized in that, Based on the channel sensitivity of each element in the weight matrix corresponding to the network channel, the elements in the weight matrix are rearranged and partitioned to obtain multiple weight sub-matrices, including: Based on the channel sensitivity of the output channel corresponding to each element in the weight matrix, the rows in the weight matrix are sorted in descending order of channel sensitivity from largest to smallest. Also, based on the channel sensitivity of the input channel corresponding to each element in the weight matrix, the columns in the weight matrix are sorted in descending order of channel sensitivity from largest to smallest, resulting in a rearranged weight matrix. The rearranged weight matrix is ​​divided into multiple blocks to obtain multiple weight submatrices.

8. The method according to any one of claims 1-6, characterized in that, The allocation of corresponding quantization resources to each weight submatrix based on the channel sensitivity of each element in the network channel includes: For each weight submatrix, the saliency of each element in each weight submatrix is ​​determined based on the channel sensitivity of the input channel and the channel sensitivity of the output channel corresponding to each element in each weight submatrix. The significance of each weight submatrix is ​​determined based on the significance of multiple elements in each weight submatrix; Based on the significance of each weight submatrix, corresponding quantization resources are allocated to each weight submatrix.

9. A model reasoning method, characterized in that, include: In response to receiving a visual language task, the visual language model is invoked to process the natural language instructions and visual information in the visual language task, and the task processing result is obtained. The target quantization matrix obtained by the visual language model based on the model quantization method described in any one of claims 1-8 is pre-deployed to the hardware device.

10. A model quantization device, characterized in that, include: An evaluation module is used to evaluate the sensitivity of each network channel of the visual language model based on the weight matrix of the visual language model, and obtain the channel sensitivity of each network channel. The weight matrix includes elements corresponding to multiple network channels. The partitioning module is used to rearrange and partition the elements in the weight matrix based on the channel sensitivity of the network channels corresponding to each element in the weight matrix, so as to obtain multiple weight submatrices. The allocation module is used to allocate corresponding quantization resources to each weight submatrix based on the channel sensitivity of the network channel corresponding to each element in each weight submatrix. The determination module is used to quantize each weight submatrix based on the quantization resources corresponding to each weight submatrix, and obtain multiple quantized submatrixes. The correction module is used to correct the plurality of quantization sub-matrices to obtain the target quantization matrix of the weight matrix, so as to deploy the visual language model to the hardware device to perform visual language tasks based on the target quantization matrix.

11. A model reasoning device, characterized in that, include: The calling module is used to respond to a received visual language task by calling a visual language model to process the natural language instructions and visual information in the visual language task and obtain the task processing result. The target quantization matrix obtained by the visual language model based on the model quantization method described in any one of claims 1-8 is pre-deployed to the hardware device.

12. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-9.

13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-9.

14. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-9.

Images