A quantitative reasoning acceleration method and system for Grounding DINO

Abstract

The invention provides a quantization reasoning acceleration method and system for Grounding DINO, belonging to the field of computer network information technology. The method comprises: replacing a Linear layer in a Grounding DINO model to be quantized with a W4Linear layer, and obtaining the weight of the Grounding DINO model to be quantized and the shape of a weight matrix; dynamically adjusting the group size according to the shape of the weight matrix, and quantizing the weight based on the group size; compressing the quantized weight, inputting the data to be detected into the Grounding DINO model after quantization and compression, and the model inferring a result based on W4A32 row-by-row dequantization matrix multiplication; solving the problem that the Grounding DINO model cannot use weight-only group quantization when deployed on a resource-constrained device, and solving the problem that the model cannot directly apply a low-bit matrix multiplication kernel.

Description

Technical Field

[0001] The present invention relates to the field of computer network information technology, and in particular to a quantitative reasoning acceleration method and system for Grounding DINO. Background Art

[0002] Grounding DINO is a Transformer-based open-set object detection model that can detect arbitrary objects specified by human language input in images. This task has wide applications due to its great potential as a general-purpose object detector. However, its Transformer-based architecture leads to a large amount of memory consumption and memory access overhead, which hinders their deployment on resource-constrained devices and increases energy consumption. To this end, it is necessary to compress this pre-trained model. Quantization is one of the most effective ways to compress pre-trained models. Quantization is the process of quantizing FP32-type values ​​into integers with low bit width, which reduces the model's memory consumption and memory access overhead.

[0003] Quantization methods are often divided into Quantization-Aware Training (QAT) and Post-training Quantization (PTQ). Although QAT can reduce the decline in model accuracy, it requires more training data, longer optimization time, and more sophisticated hyperparameter control during training. In particular, Grounding DINO requires more categories of image data during training, so QAT cannot solve these problems very friendly. Among them, PTQ includes dynamic quantization, static quantization, and weight-only quantization. Dynamic quantization means that the quantization factor of activation is calculated in real time during the inference process, which may lead to high computational overhead; while static quantization converts the weights and activation values ​​of the model to low precision before deployment, resulting in low flexibility to input changes. In contrast, weight-only quantization does not require complex quantization parameter adjustment or calibration steps, which simplifies the quantization process. Since the activation remains in float32 format, weight-only quantization provides higher flexibility and can adapt to different input values, ensuring the generalization performance of the pre-trained model.

[0004] At present, many computer vision research works focus on post-training quantization techniques, which often quantize both weights and activations. The paper "RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers" (Zhikai Li et al., Institute of Automation, Chinese Academy of Sciences, etc., December, 2022) decouples the quantization and inference processes, using a complex quantizer in the quantization stage and a simplified quantizer with scale reparameterization in the inference stage, ensuring the accuracy of quantization and the efficiency of inference. The paper "NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers" (Yijiang Liu et al., Nanjing University, etc., November, 2022) found that for a specific quantizer, adding a fixed uniform noise bias to the quantized value can theoretically significantly reduce the quantization error, and enhance the post-training activation quantization performance of the visual transformer by introducing a uniform noise bias. However, although these works have made great efforts in quantizing activations, they need to rely on specific datasets to maintain model performance, and these works do not design corresponding kernels to achieve acceleration.

[0005] Weight-only quantization has been widely studied in the field of large language models (LLMs), which confirms the effectiveness of weight-only quantization. In order to maintain the versatility of the model, the paper "AWQ: Activation-Aware Weight Quantization for On-Device LLM Compression and Acceleration" (Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang and Song Han, June, 2023) proposed activation-aware weight quantization, which mainly looks for significant weights by observing activation values, and then scales the significant weights to protect them. AWQ does not rely on any backpropagation or reconstruction, so it can well preserve the generalization ability of LLM in different domains and modalities without overfitting the calibration set. AWQ implements an efficient and flexible reasoning framework TinyChat, which achieves an acceleration of more than 3 times, but it sets a fixed value of 128 on the group size. For some Linear layers in the Grounding DINO model, the row dimension is not an integer multiple of 128, which makes it impossible to directly apply the weight group quantization method with a fixed group size. In addition, the current LLMs weight storage type is FP16, while the Grounding DINO weight storage type is FP32. The existing low-bit matrix multiplication kernel designed for LLM cannot be directly applied to the Grounding DINO model. Summary of the invention

[0006] The present invention aims to solve two key technical problems encountered when the Grounding DINO model is deployed on resource-constrained devices: first, when applying the weight-only grouping quantization method, larger fixed grouping values ​​cannot divide the row dimensions of some Linear layers, while smaller fixed values ​​will result in larger additional overhead; second, the existing low-bit matrix multiplication kernel is incompatible with the model.

[0007] The present invention solves the above technical problems through the following technical solutions: a quantitative reasoning acceleration method for Grounding DINO, the method comprising:

[0008] Step 1: Replace the Linear layer in the Grounding DINO model to be quantized with the W4Linear layer, and obtain the weights and weight matrix shape of the Grounding DINO model to be quantized;

[0009] Step 2: dynamically adjust the group size according to the shape of the weight matrix, quantize the weight based on the group size, and obtain the quantized weight;

[0010] Step 3: compress the quantized weights to obtain a quantized and compressed Grounding DINO model;

[0011] Step 4: Input the data to be tested into the quantized and compressed Grounding DINO model. The model infers the result based on the W4A32 row-by-row dequantization matrix multiplication.

[0012] Preferably, the process of replacing the Linear layer in the Grounding DINO model to be quantized with the W4Linear layer in step 1 includes:

[0013] Step 1.1, input the Grounding DINO model and prefix string to be quantized;

[0014] Step 1.2, traverse all submodules in the Grounding DINO model to be quantized and construct a full name for each submodule;

[0015] Step 1.3: Determine whether each submodule is a replaceable Linear layer. If so, record the shape of the weight matrix of the submodule. If not, return to step 1.2.

[0016] Preferably, in step 1.3, determining whether each sub-module is a replaceable Linear layer is performed by determining whether the sub-module is a Linear layer and whether the number of rows and columns of its weight matrix are not less than 128. When the number of rows and columns of the weight matrix of the sub-module are not less than 128, the sub-module is a replaceable Linear layer.

[0017] Preferably, dynamically adjusting the grouping size according to the shape of the weight matrix in step 2 is to determine the grouping size according to the row dimension of the shape of the weight matrix.

[0018] Preferably, the process of quantizing the weight based on the group size in step 2 includes:

[0019] Step 2.1, calculate the scaling factor based on the maximum and minimum values ​​in a set of data. The calculation formula is:

[0020] (1)

[0021] In formula (1), is the scaling factor for a set of data, , Respectively represent the maximum and minimum values ​​in a set of data. , They represent the maximum value 15 and the minimum value 0 of the quantized range respectively;

[0022] Step 2.2: Calculate the zero value based on the scaling factor , the calculation formula is:

[0023] (2)

[0024] Step 2.3: Calculate the quantized weights based on the scaling factor and zero value , the calculation formula is:

[0025] (3).

[0026] Preferably, in step 3, the quantized weights are compressed, mainly by using a bit shift operation to compress the four quantized weight values ​​into a 16-bit integer value, placing the first quantized weight value in the highest 4 bits of uint16, and then shifting the second quantized weight value left by 8 bits and placing it in the next 4 bits of uint16, and so on, until all four quantized weight values ​​are placed in a uint16 value.

[0027] Preferably, step 4 comprises:

[0028] Step 4.1, divide the activation matrix A (M, K) into blocks to obtain block matrices A11, A12, A21, A22, divide the weight matrix B (M, K / 4) into blocks to obtain block matrices B11, B12, B21, B22, and load each block matrix from the global memory to the shared memory one by one to obtain the activation matrix smemA and the weight matrix smemB;

[0029] Step 4.2, decompress and dequantize the weight matrix smemB in the shared memory, load the result into the register using the float4 data type, and load the activation matrix smemA in the shared memory into the register using the float4 data type;

[0030] Step 4.3, use row-by-row matrix multiplication to calculate the result matrix C, and write the elements of the result matrix C back to the global memory. The calculation formula of the result matrix C is:

[0031] (4).

[0032] The present invention also provides a quantized inference acceleration system for Grounding DINO, the system comprising:

[0033] The layer replacement module is used to replace the Linear layer in the Grounding DINO model to be quantized with the W4Linear layer, and obtain the weight and weight matrix shape of the Grounding DINO model to be quantized;

[0034] Only the weight dynamic group quantization module is used to dynamically adjust the group size according to the shape of the weight matrix, quantize the weight based on the group size, and obtain the quantized weight;

[0035] The weight compression module is used to compress the quantized weights to obtain a quantized and compressed GroundingDINO model;

[0036] The inference module is used to input the data to be tested into the quantized and compressed Grounding DINO model. The model infers the result based on the W4A32 row-by-row dequantization matrix multiplication.

[0037] Preferably, the process of replacing the Linear layer in the Grounding DINO model to be quantized with the W4Linear layer in the layer replacement module includes:

[0038] Step 1.1, input the Grounding DINO model and prefix string to be quantized;

[0039] Step 1.2, traverse all submodules in the Grounding DINO model to be quantized and construct a full name for each submodule;

[0040] Step 1.3: Determine whether each submodule is a replaceable Linear layer. If so, record the shape of the weight matrix of the submodule. If not, return to step 1.2.

[0041] Preferably, in step 1.3, determining whether each sub-module is a replaceable Linear layer is performed by determining whether the sub-module is a Linear layer and whether the number of rows and columns of its weight matrix are not less than 128. When the number of rows and columns of the weight matrix of the sub-module are not less than 128, the sub-module is a replaceable Linear layer.

[0042] Preferably, the dynamic adjustment of the grouping size according to the shape of the weight matrix in the weight-only dynamic grouping quantization module is to determine the grouping size according to the row dimension of the shape of the weight matrix.

[0043] Preferably, the process of quantizing the weight based on the group size in the weight-only dynamic grouping quantization module includes:

[0044] Step 2.1, calculate the scaling factor based on the maximum and minimum values ​​in a set of data. The calculation formula is:

[0045] (1)

[0046] In formula (1), is the scaling factor for a set of data, , Respectively represent the maximum and minimum values ​​in a set of data. , They represent the maximum value 15 and the minimum value 0 of the quantized range respectively;

[0047] Step 2.2: Calculate the zero value based on the scaling factor , the calculation formula is:

[0048] (2)

[0049] Step 2.3: Calculate the quantized weights based on the scaling factor and zero value , the calculation formula is:

[0050] (3).

[0051] Preferably, the weight compression module compresses the quantized weights, mainly using a bit shift operation to compress the four quantized weight values ​​into a 16-bit integer value, placing the first quantized weight value in the highest 4 bits of uint16, and then shifting the second quantized weight value left by 8 bits and placing it in the next 4 bits of uint16, and so on, until all four quantized weight values ​​are placed in a uint16 value.

[0052] Preferably, the process of inferring the result of the quantized and compressed Grounding DINO model based on W4A32 row-by-row dequantization matrix multiplication in the inference module includes:

[0053] Step 4.1, divide the activation matrix A (M, K) into blocks to obtain block matrices A11, A12, A21, A22, divide the weight matrix B (M, K / 4) into blocks to obtain block matrices B11, B12, B21, B22, and load each block matrix from the global memory to the shared memory to obtain the activation matrix smemA and the weight matrix smemB;

[0054] Step 4.2, decompress and dequantize the weight matrix smemB in the shared memory, load the result into the register using the float4 data type, and load the activation matrix smemA in the shared memory into the register using the float4 data type;

[0055] Step 4.3, use row-by-row matrix multiplication to calculate the result matrix C, and write the elements of the result matrix C back to the global memory. The calculation formula of the result matrix C is:

[0056] (4).

[0057] The advantages provided by the present invention are:

[0058] (1) The existing weight-only grouping quantization usually fixes the grouping size to a specific value (e.g., 128). Since the row dimensions of some Linear layers in the Grounding DINO model may not be multiples of 128, the existing weight-only grouping quantization method may need to reduce the grouping size when processing the Grounding DINO model, which will introduce greater additional overhead. In the process of quantizing weights, the present invention evaluates the row dimension size of the current quantized Linear layer in real time and dynamically adjusts the grouping size according to this size. No additional data is required for calibration and training, which not only reduces the additional overhead introduced by improper grouping, but also ensures that the performance and efficiency of the model are optimized. Through this intelligent grouping quantization technology, the Grounding DINO model can achieve higher computational efficiency and lower resource consumption while maintaining its versatility. The present invention also considers that the existing low-bit matrix multiplication kernel cannot be directly applied to the Grounding DINO model. In the inference stage, the W4A32 row-by-row dequantization matrix multiplication is used to convert the space saved by model quantization into actual speed improvement, thereby improving the inference speed of the model.

[0059] (2) Among the existing data types, there is no direct support for unsigned four-bit integers (uint4). Generally, the quantized weights need to be stored in a smaller data type to reduce the model size. The traditional approach is to put 8 uint4 values ​​into a uint32 through a bit shift operation. However, the present invention considers that in the dequantization matrix, the 4 float32 values ​​after decompression and inverse quantization need to be put into a float4. Therefore, during compression, the 4 uint4 values ​​are compressed into a uint16 through a bit shift operation.

[0060] (3) The present invention adopts a row-by-row approach to design the dequantized matrix multiplication kernel. In the CUDA kernel, the block matrices of the input activation matrix A and the weight matrix B are first loaded into the shared memory. Then, in order to further improve the computation-to-memory access ratio, the float4 data type is introduced, so that when each thread is responsible for calculating one element of the result matrix C, the memory access is 4 times faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of a quantized inference acceleration method for Grounding DINO provided in an embodiment of the present invention;

[0062] Figure 2 A flowchart of a quantized inference acceleration method for Grounding DINO provided by an embodiment of the present invention;

[0063] Figure 3A schematic diagram of compressing quantized weights in a quantized inference acceleration method for Grounding DINO provided by an embodiment of the present invention;

[0064] Figure 4 A schematic diagram of decompressing and dequantizing a weight matrix in a quantized inference acceleration method for Grounding DINO provided by an embodiment of the present invention;

[0065] Figure 5 A schematic diagram of a quantized inference acceleration system for Grounding DINO provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Example 1

[0068] like Figure 1 and Figure 2 As shown, this embodiment provides a quantized reasoning acceleration method for Grounding DINO, including:

[0069] Step 1: Load the neural network model. In this embodiment, the Grounding DINO model to be quantized is used. The Linear layer in the Grounding DINO model to be quantized is replaced with the W4Linear layer through the function replace_linear_modules. During the replacement of the Linear layer, the weight (weight) and the shape (shape) of the weight matrix of the Grounding DINO model to be quantized are obtained.

[0070] Step 1.1, input the Grounding DINO model and prefix string to be quantized;

[0071] Step 1.2, traverse all submodules in the Grounding DINO model to be quantized and construct a full name for each submodule;

[0072] Step 1.3: Determine whether each submodule is a replaceable Linear layer (nn.Linear) by judging whether it is a Linear layer and whether the number of rows and columns of its weight matrix are not less than 128. When the number of rows and columns of the weight matrix of the submodule is not less than 128, the submodule is a replaceable Linear layer. In this case, the shape of the weight matrix of the submodule is recorded, and the Linear layer is replaced with a custom Linear layer (such as W4Linear) to achieve weight quantization and compression, thereby reducing memory access and storage requirements during model inference. For submodules that do not meet the conditions, return to step 1.2 and recursively apply the above steps to ensure that all relevant Linear layers in the entire neural network module are optimized.

[0073] Step 2: Dynamically adjust the group size (Groupsize) according to the dimension of each row of the shape (shape) of the weight matrix. For example, the Groupsize is set according to the current size of each row of the weight matrix. When the size of a row of the weight matrix is ​​greater than or equal to 1024, set Groupsize=128, when the size of a row of the weight matrix is ​​greater than or equal to 256, set Groupsize=64, and when the size of this row is less than 256, set Groupsize=32. At the same time, the Groupsize can be dynamically adjusted according to the quantized accuracy.

[0074] The weight is quantized based on the group size to obtain the quantized weight, which specifically includes the following calculation process:

[0075] Step 2.1, calculate the scaling factor based on the maximum and minimum values ​​in a set of data. The calculation formula is:

[0076] (1)

[0077] In formula (1), is the scaling factor for a set of data, , Respectively represent the maximum and minimum values ​​in a set of data. , They represent the maximum value 15 and the minimum value 0 of the quantized range respectively;

[0078] Step 2.2: Calculate the zero value based on the scaling factor , the calculation formula is:

[0079] (2)

[0080] Step 2.3: Calculate the quantized weights based on the scaling factor and zero value , the zero point is used to adjust the offset of the quantized value, and the scaling factor is used to adjust the scaling ratio of the quantized value. The calculation formula is:

[0081] (3)

[0082] In traditional group quantization methods, weights are usually grouped by row, and the group size (Groupsize) is often fixed to 128. This approach may encounter challenges when processing Linear layers, because the row dimension of the Linear layer is not always an integer multiple of 128. In addition, as the group size decreases, the additional computational overhead will also increase, which may have an adverse effect on the overall efficiency of the model. In order to effectively solve these problems, the present invention proposes a dynamic group quantization method, which can intelligently adjust the group size (Groupsize) according to the actual size of the Linear layer row dimension to achieve the optimal quantization effect.

[0083] Step 3: Compress the quantized weights to obtain a quantized and compressed Grounding DINO model.

[0084] The quantized weights were obtained in step 2.3 , and the numerical range of weights is limited to between 0 and 15. In order to compress the values ​​of 4 quantized weights into a 16-bit integer (uint16), you first need to determine the position of each weight value in the final compressed value. Since each weight value ranges from 0 to 15, 4 bits can be used to represent each weight value. Starting with the first weight value, place it in the highest 4 bits of uint16, then shift the second weight value left 8 bits and place it in the next 4-bit position, and so on, until all 4 weight values ​​are placed in a uint16 value. In this way, each weight value occupies 4 bits of uint16, and a total of 16 bits are used. The specific process is as follows Figure 3 shown.

[0085] In the existing data types, there is no direct support for unsigned four-bit integers (uint4). Usually, the quantized weights need to be stored in a smaller data type to reduce the model size. The traditional approach is to put 8 uint4 values ​​into a uint32 through a bit shift operation. However, the present invention considers that in the dequantization matrix, the 4 float32 after decompression and dequantization need to be put into a float4. For this reason, when compressing, the 4 uint4 values ​​are compressed into a uint16 through a bit shift operation.

[0086] Step 4: Input the data to be tested into the quantized and compressed Grounding DINO model. The model infers the result based on the W4A32 row-by-row dequantization matrix multiplication.

[0087] Step 4.1, block the activation matrix A (M, K) to obtain block matrices A11, A12, A21, A22, block the weight matrix B (M, K / 4) to obtain block matrices B11, B12, B21, B22, and load each block matrix from the global memory to the shared memory to obtain the activation matrix smemA and the weight matrix smemB. In the memory type of CUDA, there is a shared memory (SharedMemory), which means the memory shared by all threads in the thread block. The access speed of this memory is faster, so the present invention performs matrix block division.

[0088] Step 4.2, decompress and dequantize the weight matrix smemB in the shared memory, load the result into the register using the float4 data type, and load the activation matrix smemA in the shared memory into the register using the float4 data type;

[0089] Step 4.3, use row-by-row matrix multiplication to calculate the result matrix C. At this time, define a macro function, which is mainly used for float4 data type calculation. In this way, four multiplication operations and three addition operations can be performed each time the memory is accessed. Finally, the elements of the result matrix C are written back to the global memory. The calculation formula of the result matrix C is:

[0090] (4)

[0091] The present invention designs the W4A32 row-by-row dequantization matrix multiplication kernel, which mainly adopts the row-by-row method to design the dequantization matrix multiplication kernel. In the CUDA kernel, the block matrices of the input activation matrix A and the weight matrix B are first loaded into the shared memory. Then, in order to further improve the calculation memory access ratio, the float4 data type is introduced, so that when each thread is responsible for calculating an element of the result matrix C, the memory access is 4 times faster. By the row-by-row method, the parallelism of the dequantization matrix multiplication is improved.

[0092] Example 2

[0093] like Figure 5 As shown, this embodiment provides a quantized inference acceleration system for Grounding DINO, including:

[0094] The layer replacement module is used to replace the Linear layer in the Grounding DINO model to be quantized with the W4Linear layer, and obtain the weights and weight matrix shape of the Grounding DINO model to be quantized.

[0095] The process of replacing the Linear layer in the Grounding DINO model to be quantized with the W4Linear layer in the layer replacement module includes:

[0096] Step 1.1: Input the Grounding DINO model and prefix string to be quantized.

[0097] Step 1.2: Traverse all submodules in the Grounding DINO model to be quantized and construct a full name for each submodule.

[0098] Step 1.3: Determine whether each submodule is a Linear layer and whether the number of rows and columns of its weight matrix are not less than 128. If the number of rows and columns of the weight matrix of the submodule is not less than 128, the submodule is a replaceable Linear layer, and the shape of the weight matrix of the submodule is recorded. If not, return to step 1.2.

[0099] The weight-only dynamic grouping quantization module is used to dynamically adjust the grouping size according to the dimension of each row of the shape of the weight matrix, quantize the weight based on the grouping size, and obtain the quantized weight.

[0100] The process of quantizing weights based on group size includes:

[0101] Step 2.1, calculate the scaling factor based on the maximum and minimum values ​​in a set of data. The calculation formula is:

[0102] (1)

[0103] In formula (1), is the scaling factor for a set of data, , Respectively represent the maximum and minimum values ​​in a set of data. , They represent the maximum value 15 and the minimum value 0 of the quantized range respectively.

[0104] Step 2.2: Calculate the zero value based on the scaling factor , the calculation formula is:

[0105] (2)

[0106] Step 2.3: Calculate the quantized weights based on the scaling factor and zero value , the calculation formula is:

[0107] (3)

[0108] The weight compression module is used to compress the quantized weights to obtain the quantized compressed GroundingDINO model. The four quantized weight values ​​are compressed into a 16-bit integer value by using the bit shift operation, and the first quantized weight value is placed in the highest 4 bits of uint16, and the second quantized weight value is shifted left by 8 bits and placed in the next 4 bits of uint16, and so on, until all four quantized weight values ​​are placed in a uint16 value.

[0109] The inference module is used to input the data to be tested into the quantized and compressed Grounding DINO model. The model infers the result based on the W4A32 row-by-row dequantization matrix multiplication. The specific process includes the following:

[0110] Step 4.1, divide the activation matrix A (M, K) into blocks to obtain block matrices A11, A12, A21, A22, divide the weight matrix B (M, K / 4) into blocks to obtain block matrices B11, B12, B21, B22, and load each block matrix from the global memory to the shared memory to obtain the activation matrix smemA and the weight matrix smemB.

[0111] Step 4.2, decompress and dequantize the weight matrix smemB in the shared memory, load the result into the register using the float4 data type, and load the activation matrix smemA in the shared memory into the register using the float4 data type;

[0112] Step 4.3, use row-by-row matrix multiplication to calculate the result matrix C, and write the elements of the result matrix C back to the global memory. The calculation formula of the result matrix C is:

[0113] (4)

[0114] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A quantitative inference acceleration method for Grounding DINO. Grounding DINO is an open set object detection model based on Transformer, which is used to detect any object specified by human language input in an image. It is characterized by: Methods include: S1. Replace the Linear layer in the Grounding DINO model to be quantized with the W4Linear layer, and obtain the weights and weight matrix shape of the Grounding DINO model to be quantized; including: S1.

1. Input the Grounding DINO model and prefix string to be quantized; S1.2, traverse all submodules in the Grounding DINO model to be quantized, and construct a full name for each submodule; S1.3, determine whether each submodule is a replaceable Linear layer, if so, record the shape of the weight matrix of the submodule, if not, return to S1.2; S2, dynamically adjust the group size according to the shape of the weight matrix, quantize the weight based on the group size, and obtain the quantized weight; S3. Compress the quantized weights to obtain a quantized compressed Grounding DINO model. The quantized weights are compressed by using a bit shift operation, which mainly compresses the four quantized weight values ​​into a 16-bit integer value, and places the first quantized weight value in the highest 4 bits of uint16. Then, the second quantized weight value is shifted left by 8 bits and placed in the next 4 bits of uint16, and so on, until all four quantized weight values ​​are placed in a uint16 value. S4. Input the data to be detected into the Grounding DINO model after quantization and compression. The model infers the result based on W4A32 row-by-row dequantization matrix multiplication; including: S4.

1. Divide the activation matrix A(M, K) into blocks to obtain block matrices A11, A12, A21, and A22, divide the weight matrix B(M, K / 4) into blocks to obtain block matrices B11, B12, B21, and B22, and load each block matrix from the global memory to the shared memory one by one to obtain the activation matrix smemA and the weight matrix smemB; S4.2, decompress and dequantize the weight matrix smemB in the shared memory, load the result into the register using the float4 data type, and load the activation matrix smemA in the shared memory into the register using the float4 data type; S4.

3. Calculate the result matrix C using row-by-row matrix multiplication, and write the elements of the result matrix C back to the global memory. The result matrix C is: 。 2. The quantized inference acceleration method for Grounding DINO according to claim 1, characterized in that: In S1.3, whether each submodule is a replaceable Linear layer is determined by judging whether the submodule is a Linear layer and whether the number of rows and columns of its weight matrix are not less than 128. When the number of rows and columns of the weight matrix of the submodule are not less than 128, the submodule is a replaceable Linear layer.

3. The quantized inference acceleration method for Grounding DINO according to claim 1, characterized in that: In S2, the grouping size is dynamically adjusted according to the shape of the weight matrix, and the grouping size is determined according to the row dimension of the shape of the weight matrix.

4. The quantized inference acceleration method for Grounding DINO according to claim 3, characterized in that: The process of quantizing weights based on group size in S2 includes: S2.

1. Calculate the scaling factor based on the maximum and minimum values ​​in a set of data. The calculation formula is: in, is the scaling factor for a set of data, , Respectively represent the maximum and minimum values ​​in a set of data. , They represent the maximum value 15 and the minimum value 0 of the quantized range respectively; S2.

2. Calculate the zero value based on the scaling factor , the calculation formula is: S2.

3. Calculate the quantized weight based on the scaling factor and zero value , the calculation formula is: 。 5. A quantitative inference acceleration system for Grounding DINO, which is an open-set object detection model based on Transformer, used to detect arbitrary objects specified by human language input in images, characterized by: The system includes: The layer replacement module is used to replace the Linear layer in the Grounding DINO model to be quantized with the W4Linear layer, and obtain the weight and weight matrix shape of the Grounding DINO model to be quantized; including: S1.

1. Input the Grounding DINO model and prefix string to be quantized; S1.2, traverse all submodules in the Grounding DINO model to be quantized, and construct a full name for each submodule; S1.3, determine whether each submodule is a replaceable Linear layer, if so, record the shape of the weight matrix of the submodule, if not, return to S1.2; Only the weight dynamic group quantization module is used to dynamically adjust the group size according to the shape of the weight matrix, quantize the weight based on the group size, and obtain the quantized weight; The weight compression module is used to compress the quantized weights to obtain the quantized compressed Grounding DINO model. The quantized weights are compressed by mainly using the bit shift operation to compress the 4 quantized weight values ​​into a 16-bit integer value, placing the first quantized weight value in the highest 4 bits of uint16, and then shifting the second quantized weight value left by 8 bits and placing it in the next 4 bits of uint16, and so on, until all 4 quantized weight values ​​are placed in a uint16 value. The inference module is used to input the data to be tested into the quantized and compressed Grounding DINO model. The model infers the result based on the W4A32 row-by-row dequantization matrix multiplication; it includes: S4.

1. Divide the activation matrix A(M, K) into blocks to obtain block matrices A11, A12, A21, and A22, divide the weight matrix B(M, K / 4) into blocks to obtain block matrices B11, B12, B21, and B22, and load each block matrix from the global memory to the shared memory to obtain the activation matrix smemA and the weight matrix smemB; S4.2, decompress and dequantize the weight matrix smemB in the shared memory, load the result into the register using the float4 data type, and load the activation matrix smemA in the shared memory into the register using the float4 data type; S4.

3. Calculate the result matrix C using row-by-row matrix multiplication, and write the elements of the result matrix C back to the global memory. The result matrix C is: 。 6. The quantized inference acceleration system for Grounding DINO according to claim 5, characterized in that: In S1.3, whether each submodule is a replaceable Linear layer is determined by judging whether the submodule is a Linear layer and whether the number of rows and columns of its weight matrix are not less than 128. When the number of rows and columns of the weight matrix of the submodule are not less than 128, the submodule is a replaceable Linear layer.

7. The quantized inference acceleration system for Grounding DINO according to claim 5, characterized in that: In the weight-only dynamic grouping quantization module, the grouping size is dynamically adjusted according to the shape of the weight matrix, and the grouping size is determined according to the row dimension of the shape of the weight matrix.

8. The quantized inference acceleration system for Grounding DINO according to claim 7, characterized in that: The process of quantizing weights based on group size in the weight-only dynamic group quantization module includes: S2.

1. Calculate the scaling factor based on the maximum and minimum values ​​in a set of data. The calculation formula is: in, is the scaling factor for a set of data, , Respectively represent the maximum and minimum values ​​in a set of data. , They represent the maximum value 15 and the minimum value 0 of the quantized range respectively; S2.

2. Calculate the zero value based on the scaling factor , the calculation formula is: S2.

3. Calculate the quantized weight based on the scaling factor and zero value , the calculation formula is: 。

Images