Inference acceleration method and device applied to edge device and electronic device

Abstract

The application provides an inference acceleration method and device applied to an edge device and an electronic device. The application divides a weight matrix of a pre-training model into N weight sub-blocks, divides an activation matrix of the pre-training model into M activation sub-blocks, quantizes the weight sub-blocks and the activation sub-blocks corresponding to the weight sub-blocks in the pre-training model, and obtains a target model. If the quantization bit widths of at least two weight sub-blocks are the same, the weight values of the at least two weight sub-blocks are quantized based on weight value quantization hyperparameters corresponding to the at least two weight sub-blocks and the activation values of the activation sub-blocks corresponding to each weight sub-block are quantized based on activation value quantization hyperparameters corresponding to the activation sub-blocks, and the at least two weight sub-blocks are quantized as a whole. The weight sub-blocks with the same quantization bit width and the corresponding activation sub-blocks are processed as a whole. When the sub-blocks are loaded, the memory access mode is changed from random jumping to sequential reading and writing. The weight values and the activation values are quantized, the inference speed of the model is improved, and the storage space occupied by the model is reduced.

Description

Technical Field

[0001] This application relates to the field of model training technology, and in particular to inference acceleration methods, devices and electronic devices applied to edge devices. Background Technology

[0002] In real-world scenarios, some models, such as large language models, have been widely used in fields such as natural language processing and computer vision. However, these models are large in scale, consume a lot of storage, and have high computational costs, which makes their overall inference speed slow and unable to be deployed properly on some devices with limited performance, such as edge devices. Summary of the Invention

[0003] In view of this, this application provides a method, apparatus, and electronic device for accelerating inference on edge devices, so as to reduce the storage space occupied by the model and improve the inference speed of the model.

[0004] The technical solution provided in this application is as follows:

[0005] According to an embodiment of the first aspect of this application, a method for accelerating inference in an edge device is provided, the method comprising:

[0006] The weight matrix of the pre-trained model is divided into N weight sub-blocks; where N is greater than 1, and each weight sub-block has a corresponding quantization bit width.

[0007] The activation matrix of the pre-trained model is divided into M activation sub-blocks; where M is less than or equal to N; each activation sub-block is configured with a corresponding quantization bit width; each activation sub-block corresponds to at least one weight sub-block;

[0008] The weight sub-blocks and their corresponding activation sub-blocks in the pre-trained model are quantized to obtain the target model. The target model is deployed on an edge device for accelerated inference of the task. When quantizing the weight sub-blocks and their corresponding activation sub-blocks, if at least two weight sub-blocks have the same quantization bit width, the at least two weight sub-blocks are quantized as a whole based on the quantization hyperparameters of the weight values ​​corresponding to the at least two weight sub-blocks and the quantization hyperparameters of the activation values ​​corresponding to the activation sub-blocks of each weight sub-block.

[0009] Optionally, the quantization bit width corresponding to each weighted sub-block is determined according to the following method:

[0010] The reference weight range corresponding to the weight sub-block is determined based on the weight value in the weight sub-block;

[0011] The quantization bit width corresponding to the reference weight range is determined as the quantization bit width corresponding to the weight sub-block.

[0012] Optionally, the N weight sub-blocks are of the same size; the activation matrix of the pre-trained model is divided into M activation sub-blocks, including:

[0013] Based on the size of the weight sub-blocks, the number M of the weight matrix divided along the input channel dimension is determined;

[0014] Based on M, the activation matrix corresponding to the weight matrix is ​​divided into M activation sub-blocks along the input channel dimension.

[0015] Optionally, the quantization hyperparameter of the weight value corresponding to any weighted sub-block is determined according to the following method:

[0016] The quantization interval of the weight value corresponding to the weight sub-block is determined based on the weight value in the weight sub-block and the quantization bit width corresponding to the weight sub-block.

[0017] The weight value quantization interval is used to determine the weight value quantization hyperparameter corresponding to the weight sub-block.

[0018] Optionally, the activation value quantization hyperparameter corresponding to any activation sub-block is determined according to the following method:

[0019] The activation value quantization interval corresponding to the activation sub-block is determined based on the activation value in the activation sub-block and the quantization bit width corresponding to the activation sub-block.

[0020] The activation value quantization interval is used to determine the activation value quantization hyperparameter corresponding to the activation sub-block.

[0021] Optionally, the pre-trained model includes at least one MHSA layer structure and at least one FFN layer structure. The MHSA layer structure and the FFN layer structure include multiple sorting operators, which are used to sort the weight sub-blocks based on the quantization bit width; wherein weight sub-blocks with the same quantization bit width are sorted together.

[0022] Optionally, the method further includes:

[0023] If it is found that the first sorting operator in any layer structure is used to sort the first weight matrix and the second sorting operator is used to sort the calculation result of the first weight matrix, then the first sorting operator and the second sorting operator are merged into a whole to sort the first weight matrix and the calculation result of the first weight matrix.

[0024] According to an embodiment of a second aspect of this application, a reasoning acceleration device for an edge device is provided, the device comprising:

[0025] The first partitioning unit is used to divide the weight matrix of the pre-trained model into N weight sub-blocks; where N is greater than 1, and each weight sub-block has a corresponding quantization bit width;

[0026] The second partitioning unit is used to divide the activation matrix of the pre-trained model into M activation sub-blocks; where M is less than or equal to N; each activation sub-block is configured with a corresponding quantization bit width; each activation sub-block corresponds to at least one weight sub-block;

[0027] A quantization processing unit is used to quantize the weight sub-blocks and their corresponding activation sub-blocks in the pre-trained model to obtain a target model. The target model is deployed on an edge device for accelerated inference of the task. When quantizing the weight sub-blocks and their corresponding activation sub-blocks, if at least two weight sub-blocks have the same quantization bit width, the at least two weight sub-blocks are quantized as a whole based on the quantization hyperparameters of the weight values ​​corresponding to the at least two weight sub-blocks and the quantization hyperparameters of the activation values ​​corresponding to the activation sub-blocks of each weight sub-block.

[0028] Optionally, the quantization bit width corresponding to each weighted sub-block is determined according to the following method:

[0029] The reference weight range corresponding to the weight sub-block is determined based on the weight value in the weight sub-block;

[0030] The quantization bit width corresponding to the reference weight range is determined as the quantization bit width corresponding to the weight sub-block;

[0031] And / or, the N weight sub-blocks are of the same size; the activation matrix of the pre-trained model is divided into M activation sub-blocks, including:

[0032] Based on the size of the weight sub-blocks, the number M of the weight matrix divided along the input channel dimension is determined;

[0033] Based on M, the activation matrix corresponding to the weight matrix is ​​divided into M activation sub-blocks along the input channel dimension;

[0034] And / or, the weight value quantization hyperparameter corresponding to any weighted sub-block is determined according to the following method:

[0035] The quantization interval of the weight value corresponding to the weight sub-block is determined based on the weight value in the weight sub-block and the quantization bit width corresponding to the weight sub-block.

[0036] The weight value quantization interval is used to determine the weight value quantization hyperparameter corresponding to the weight sub-block;

[0037] And / or, the activation value quantization hyperparameter corresponding to any activation sub-block is determined according to the following method:

[0038] The activation value quantization interval corresponding to the activation sub-block is determined based on the activation value in the activation sub-block and the quantization bit width corresponding to the activation sub-block.

[0039] The activation value quantization interval is used to determine the activation value quantization hyperparameter corresponding to the activation sub-block;

[0040] And / or, the pre-trained model includes at least one MHSA layer structure and at least one FFN layer structure, wherein the MHSA layer structure and the FFN layer structure include multiple sorting operators, which are used to sort the weight sub-blocks based on the quantization bit width; wherein weight sub-blocks with the same quantization bit width are sorted together;

[0041] And / or, the quantization processing unit is further configured to:

[0042] If it is found that the first sorting operator in any layer structure is used to sort the first weight matrix and the second sorting operator is used to sort the calculation result of the first weight matrix, then the first sorting operator and the second sorting operator are merged into a whole to sort the first weight matrix and the calculation result of the first weight matrix.

[0043] According to an embodiment of a third aspect of this application, an electronic device is provided, comprising: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; the processor is configured to execute the machine-executable instructions to implement the method described in the first aspect.

[0044] As can be seen from the above technical solution, this application divides the pre-trained model's weight matrix into N weight sub-blocks and the pre-trained model's activation matrix into M activation sub-blocks to quantize the weight sub-blocks and their corresponding activation sub-blocks in the pre-trained model, thereby obtaining the target model. If at least two weight sub-blocks have the same quantization bit width, then based on the quantization hyperparameters of the weight values ​​corresponding to these at least two weight sub-blocks and the quantization hyperparameters of the activation values ​​corresponding to the activation sub-blocks of each weight sub-block, these at least two weight sub-blocks are quantized as a whole. Specifically, by processing weight sub-blocks and their corresponding activation blocks with the same quantization bit width as a whole, the memory access mode changes from random jumps to sequential read / write when loading these sub-blocks, and both weight values ​​and activation values ​​are quantized simultaneously, improving the model's inference speed and reducing the model's storage space usage. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the principles of this application.

[0046] Figure 1 A flowchart illustrating a method for accelerating inference on edge devices provided in an embodiment of this application;

[0047] Figure 2 This application provides a schematic diagram of weighted sub-block partitioning and sorting.

[0048] Figure 3 A schematic diagram of the model structure and sorting operator provided in the embodiments of this application;

[0049] Figure 4 This is a schematic diagram of the inference process for an abnormal behavior monitoring system under conventional methods provided in the embodiments of this application.

[0050] Figure 5 This is a schematic diagram of the inference process for an abnormal behavior monitoring system after quantification parameters, provided in an embodiment of this application.

[0051] Figure 6 A structural diagram of an inference acceleration device applied to an edge device is provided in an embodiment of this application;

[0052] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0054] In real-world scenarios, some models, such as large language models, have been widely used in fields such as natural language processing and computer vision. However, these models are large in scale, consume a lot of storage, and have high computational costs, which makes it impossible to deploy them properly on some devices with limited performance, such as edge devices.

[0055] Based on this, the commonly used method is to compress the model before deployment. Compression techniques include sparsity, quantization, distillation, and low-rank decomposition.

[0056] The following section mainly introduces the quantification methods.

[0057] In relevant schemes, the quantization method for weight values ​​is usually per-channel quantization.

[0058] The weight matrix W can be represented as Cout*Cin, where Cout represents the number of output channels and Cin represents the number of input channels. Perchannel quantization means that each output channel corresponds to a different quantization hyperparameter.

[0059] The quantization method for activation values ​​is usually per-tensor quantization.

[0060] The activation matrix X can be represented as K*Cin, where K represents the number of samples participating in a single operation and Cin represents the number of input channels. The entire activation matrix X typically has only one quantization hyperparameter.

[0061] However, existing quantization methods still suffer from high storage consumption and computational costs. This is especially problematic when deploying pre-trained models to devices with limited performance, such as edge devices, where the edge devices themselves may not be able to support the deployment of the pre-trained models.

[0062] Based on this, this application proposes a reasoning acceleration method for edge devices to reduce storage footprint and computational cost.

[0063] Please refer to Figure 1 , Figure 1 A flowchart illustrating an inference acceleration method for edge devices provided in an embodiment of this application.

[0064] As one embodiment, this method can be applied to controllers in electronic devices, such as central processing units (CPUs), and this application does not limit it.

[0065] It should be noted that the pre-trained model can include multiple layers, and each layer can include one or more weight matrices, and each weight matrix has a corresponding activation matrix. The processing of the weight matrix and activation matrix in this application can be performed on any weight matrix in any layer and the activation matrix corresponding to that weight matrix.

[0066] like Figure 1 As shown, the method may include the following steps:

[0067] Step 101: Divide the weight matrix of the pre-trained model into N weight sub-blocks.

[0068] Where N is an integer greater than 1, and each weighted sub-block is configured with a corresponding quantization bit width.

[0069] In this embodiment, the weight matrix of the pre-trained model can be obtained directly after pre-training is completed. The weight matrix can be a matrix in the format of Cout*Cin.

[0070] As an example, the N weight sub-blocks can have the same size. The process of dividing the weight matrix into N weight sub-blocks can be based on the dimension of the output channel, that is, dividing from the Cout dimension without breaking the arrangement of Cin, or dividing according to a specified size. Similarly, the N weight sub-blocks can also have different sizes, and this application does not limit this.

[0071] In this embodiment, each weighted sub-block is configured with a corresponding quantization bit width. The quantization bit width refers to the number of bits used to represent a value. For example, a quantization bit width of 8 bits means that each weight value in the weighted sub-block is represented by an 8-bit binary number. The lower the quantization bit width, the higher the data compression rate, but the greater the risk of precision loss.

[0072] As an example, the quantization bit width corresponding to each weighted sub-block can be determined according to the following method:

[0073] Determine the reference weight range corresponding to the weight sub-block based on the weight values ​​in the weight sub-block;

[0074] The quantization bit width corresponding to the reference weight range is determined as the quantization bit width corresponding to the weight sub-block.

[0075] In this embodiment, the correspondence between weight ranges and quantization bit widths can be preset. Considering that weighted sub-blocks with higher weight values ​​are more important and require higher precision, a higher quantization bit width can be set for high weight ranges, allowing these sub-blocks to be configured with larger quantization bit widths. For example, a weight range of 0 to 0.6 corresponds to a 4-bit quantization bit width, and a weight range of 0.6 to 1 corresponds to an 8-bit quantization bit width. The weight ranges and quantization bit widths can be adjusted according to actual needs, and this application does not impose any limitations on this.

[0076] In addition, the choice of quantization bit width can be based on the size of the processing cores supported in the edge device. For example, if the edge device includes three types of processing cores: 8*8, 8*4, and 4*4 (for example, an 8*8 processing core means that the processing core is used to perform operations on two matrices with a quantization bit width of 8 bits), then the weighted sub-block can be configured with either 8-bit or 4-bit quantization bit widths.

[0077] One specific method for determining the reference weight range corresponding to a weighted sub-block based on its weight values ​​could be:

[0078] This application does not impose any restrictions on the determination of the maximum absolute value of the weight values ​​in the weighted sub-block and the reference weight range to which the maximum value belongs; or, the average absolute value of the weight values ​​in the weighted sub-block and the reference weight range to which the average value belongs.

[0079] This concludes the description of step 101. We will now proceed to step 102.

[0080] Step 102: Divide the activation matrix of the pre-trained model into M activation sub-blocks.

[0081] Where M is less than or equal to N; each activation sub-block is configured with a corresponding quantization bit width; any activation sub-block corresponds to at least one weight sub-block.

[0082] In this embodiment, the activation matrix refers to the activation matrix corresponding to the weight matrix in step 101, which is obtained by inputting sample data into the training model.

[0083] As an example, when the N weight sub-blocks are of the same size, a specific method for dividing the activation matrix of a pre-trained model into M activation sub-blocks may include:

[0084] Based on the size of the weight sub-blocks, determine the number M of divisions the weight matrix will be in the input channel dimension; based on M, divide the activation matrix corresponding to the weight matrix into M activation sub-blocks in the input channel dimension.

[0085] Specifically, the number M of weight matrix divisions along the input channel dimension can be determined based on the size of the weight sub-blocks. For example, if the size of the weight matrix is ​​30*20, i.e., Cout=30, Cin=20, and the size of the weight sub-blocks is 3*2, it means that the weight matrix is ​​divided into 100 weight sub-blocks, and the number M of divisions along the input channel dimension is 20 / 2=10.

[0086] After determining the number M of partitions along the input channel dimension, the activation matrix can be further divided into M activation sub-blocks along the input channel dimension. For example, if the size of the activation matrix is ​​2*20, it can be divided into M blocks along the input channel dimension, that is, the activation matrix is ​​divided into 10 2*2 activation sub-blocks. Here, each 2*2 activation sub-block corresponds to 10 3*2 weight sub-blocks.

[0087] In this embodiment, each activation sub-block is also configured with a corresponding quantization bit width. The quantization bit widths corresponding to each activation sub-block may be the same or different. The quantization bit width configured for each activation sub-block may be the same or different from the quantization bit width configured for the weight sub-block corresponding to that activation sub-block. This application does not impose any restrictions on this.

[0088] This concludes the description of step 102. We will now proceed to step 103.

[0089] Step 103: Quantize the weight sub-blocks and their corresponding activation sub-blocks in the pre-trained model to obtain the target model.

[0090] Specifically, when quantizing a weighted sub-block and its corresponding activation sub-block, if at least two weighted sub-blocks have the same quantization bit width, the at least two weighted sub-blocks are quantized as a whole based on the quantization hyperparameters of the weight values ​​corresponding to the at least two weighted sub-blocks and the quantization hyperparameters of the activation values ​​corresponding to the activation sub-blocks of each weighted sub-block.

[0091] In this embodiment, the quantization bit width of the weighted sub-blocks can be used as a benchmark to quantize each weighted sub-block with the same quantization bit width and the corresponding activation sub-blocks of each weighted sub-block as a whole, so as to improve the quantization processing efficiency.

[0092] The weight quantization hyperparameters for each weighted sub-block and the activation value quantization hyperparameters for each activated sub-block are determined according to the following method:

[0093] Based on the weight values ​​in the weight sub-block and the corresponding quantization bit width, determine the quantization interval of the weight values ​​corresponding to the weight sub-block; determine the quantization hyperparameter of the weight values ​​corresponding to the weight sub-block based on the quantization interval of the weight values.

[0094] Based on the activation value in the activation sub-block and the quantization bit width corresponding to the activation sub-block, determine the activation value quantization interval corresponding to the activation sub-block; determine the activation value quantization hyperparameter corresponding to the activation sub-block based on the activation value quantization interval.

[0095] Specifically, in this embodiment, the weight value quantization hyperparameters corresponding to the weighted sub-blocks and the activation value quantization hyperparameters corresponding to the activation sub-blocks can be determined according to the following formulas:

[0096]

[0097] Where α is the maximum absolute value of the statistical distribution (for weighted sub-blocks, it is the maximum absolute value of the weight values ​​in the weighted sub-blocks; for activated sub-blocks, it is the maximum absolute value of the activation values ​​in the activated sub-blocks), L is the quantization bit width, and s represents the quantization interval, which is the quantization hyperparameter.

[0098] It should be noted that in this embodiment, the quantization hyperparameters are all obtained by static quantization, that is, by determining the weight value quantization hyperparameters corresponding to each weight sub-block and the activation value quantization hyperparameters corresponding to each activation sub-block through pre-obtained sample data, and then storing the above hyperparameters for use during model inference, so as to improve the running efficiency in actual deployment, accelerate inference, and reduce the consumption of hardware resources; rather than determining the weight value quantization hyperparameters corresponding to each weight sub-block and the activation value quantization hyperparameters corresponding to each activation sub-block based on the data to be detected during the actual model inference process.

[0099] As an example, a specific method for quantizing each weight sub-block with the same quantization bit width and the corresponding activation sub-block as a whole can be: sorting each weight sub-block according to the quantization bit width, and sorting weight sub-blocks with the same quantization bit width together and storing them continuously.

[0100] In this embodiment, the pre-trained model includes at least one MHSA (Multi-Head Self-Attention) layer structure and at least one FFN (Feed-Forward Network) layer structure. Both the MHSA and FFN layer structures include multiple sorting operators used to sort weight sub-blocks based on their quantization bit width; weight sub-blocks with the same quantization bit width are sorted together. The sorting operators can be used to achieve the above-mentioned process of sorting weight sub-blocks according to their quantization bit width, grouping weight sub-blocks with the same quantization bit width together, and storing them contiguously.

[0101] As an example, if it is found that a first sorting operator in any layer structure is used to sort the first weight matrix and a second sorting operator is used to sort the calculation result of the first weight matrix, then the first sorting operator and the second sorting operator are merged into a whole to sort the first weight matrix and the calculation result of the first weight matrix.

[0102] The specific process of the merge sorting operator will be described in detail below through specific embodiments, and will not be repeated here.

[0103] In this embodiment, the specific quantization method can be implemented according to the following formula:

[0104]

[0105] Where, x q This refers to the quantized value, where x is the value before quantization (for weighted sub-blocks, x is the weight value; for activated sub-blocks, x is the activation value), s is the quantization hyperparameter corresponding to the value before quantization (for weighted sub-blocks, s is the weight value quantization hyperparameter; for activated sub-blocks, s is the activation value quantization hyperparameter), and L is the quantization bit width. Indicates to The result is rounded to the nearest integer. The clip() function is a truncation function. For example, clip(a, b, c) means that a is restricted to the specified minimum value b and maximum value c. If a is less than the minimum value b, it will be truncated to the minimum value b. If a is greater than the maximum value c, it will be truncated to the maximum value c.

[0106] It should be noted that when quantizing weighted sub-blocks with the same quantization bit width and their corresponding activation sub-blocks as a whole, although the quantization bit width of each weighted sub-block is the same, the quantization bit width of the corresponding activation sub-block may not be the same as that of the weighted sub-block, and even the quantization bit widths of each activation sub-block may not be the same. In the process, the highest quantization bit width among the activation sub-blocks can be used as the benchmark for quantizing the activation sub-blocks corresponding to each weighted sub-block.

[0107] For example, a weight sub-block with a quantization width of 4 bits and its corresponding activation sub-block are processed as a whole. However, some 4-bit weight sub-blocks correspond to activation sub-blocks with a quantization width of 8 bits, and some 4-bit weight sub-blocks correspond to activation sub-blocks with a quantization width of 4 bits. In this case, an 8*4 operation core can be used to process the weight sub-blocks and their corresponding activation sub-blocks. That is, the activation sub-blocks are based on the highest quantization width of 8 bits, and the weight sub-blocks are based on a 4-bit quantization width.

[0108] As can be seen, the above method treats weight sub-blocks with the same quantization bit width as a whole, stores weight sub-blocks with the same quantization bit width in physical contiguous form, and stores the corresponding activation sub-blocks of weight sub-blocks with the same quantization bit width in physical contiguous form. When loading these sub-blocks, the memory access mode changes from random jump to sequential read and write, which greatly improves the loading speed.

[0109] After quantizing the pre-trained model to obtain the target model, the target model can be deployed on edge devices to accelerate inference for tasks.

[0110] It should be noted that when the edge device receives a sample to be detected, it can quantize the sample to be detected according to the activation value quantization hyperparameter of the activation matrix corresponding to the input layer of the pre-trained model. Based on the quantized sample to be detected, the activation value quantization hyperparameter, and the weight value quantization hyperparameter, the detection result is obtained. The detection result is then dequantized according to the activation value quantization interval corresponding to the input layer of the pre-trained model to obtain the target prediction result.

[0111] This concludes the description of step 103.

[0112] This concludes the discussion on... Figure 1 This describes a method for accelerating inference on edge devices.

[0113] This application divides the pre-trained model's weight matrix into N weight sub-blocks and its activation matrix into M activation sub-blocks. The weight sub-blocks and their corresponding activation sub-blocks in the pre-trained model are then quantized to obtain the target model. If at least two weight sub-blocks have the same quantization bit width, they are quantized as a whole based on the weight value quantization hyperparameters corresponding to these two weight sub-blocks and the activation value quantization hyperparameters corresponding to the activation sub-blocks. Different weight sub-blocks and activation sub-blocks are assigned different weight value quantization hyperparameters. By processing sub-blocks with the same quantization bit width as a whole, and simultaneously quantizing both weights and activations, the storage space and computational cost of the model are reduced.

[0114] The following is through Figures 2 to 5 This application provides a detailed description of the inference acceleration method for edge devices proposed in this application.

[0115] Please refer to Figure 2 , Figure 2 This is a schematic diagram of weighted sub-block partitioning and sorting provided in an embodiment of this application.

[0116] like Figure 2 As shown, X is a 2*6 activation matrix and W is a 4*6 weight matrix. In the weight matrix, each 1*1 block represents a weight sub-block, and different colors of the weight sub-blocks represent different quantization bit widths. Each column of the activation matrix, i.e., a 2*1 block, is an activation sub-block, which corresponds to all the weight sub-blocks included in the corresponding column of the weight matrix.

[0117] After dividing the weight sub-blocks and activation sub-blocks, the weight sub-blocks can be sorted according to the quantization bit width using the sorting operator. Weight sub-blocks with the same quantization bit width are arranged together to obtain three sets of weight sub-blocks W1, W2 and W3. The activation sub-blocks corresponding to these three sets of weight sub-blocks are X1, X2 and X3, respectively.

[0118] When quantizing the activation matrix and the weight matrix, weight sub-blocks with the same quantization bit width can be quantized as a whole according to the quantization bit width configured for weight sub-blocks W1, W2 and W3.

[0119] Specifically, the inference quantification process of a large model can be represented by the following formula:

[0120]

[0121] in The hyperparameter representing the weight values ​​of the weight matrix; The activation value quantization hyperparameter representing the activation matrix. Represents the weight values ​​in the weight matrix; This represents the activation value in the activation matrix.

[0122] In this embodiment, due to the introduction of hybrid quantization, the weight matrix is ​​split into multiple weight sub-blocks, and the activation matrix is ​​split into multiple activation sub-blocks. Each weight sub-block can be expressed with different expression precisions. Therefore, the above formula can be transformed into:

[0123]

[0124] in, The hyperparameter representing the weight value quantization of the i-th weighted sub-block; The hyperparameter representing the activation value quantization of the activation sub-block corresponding to the i-th weighted sub-block. This represents the weight value in the i-th weighted sub-block; This represents the activation value in the activation sub-block corresponding to the i-th weighted sub-block.

[0125] Since the mixed blocks are different and their precision cannot be aligned, it is often necessary to move blocks with the same precision together. That is, weighted sub-blocks with the same quantization bit width can be quantized as a whole to speed up the processing and improve the processing efficiency.

[0126] This concludes the discussion on... Figure 2 The description.

[0127] Please refer to Figure 3 , Figure 3 The diagram shows the model structure and sorting operator provided in the embodiments of this application.

[0128] like Figure 3 As shown, the pre-trained model includes at least one MHSA layer structure and at least one FFN layer structure. Figure 3 The left side shows a schematic diagram of the MHSA structure, and the right side shows a schematic diagram of the FFN structure. Both the MHSA and FFN layer structures include multiple sorting operators (the pentagrams in the diagrams represent sorting operators). The sorting operators are used to sort the weight sub-blocks based on the quantization bit width. The specific composition of the MHSA and FFN layer structures are common terms in the field of deep learning, and the specific components of each structure will not be elaborated here.

[0129] As can be seen in the FFN structure, the lower sorting operator is used to sort the weight sub-blocks in the weight matrix (denoted as the first weight matrix) at the up and gate positions, while the upper sorting operator is used to sort the calculation results of the first weight matrix. At this point, the two sorting operators in the FFN structure can be merged (i.e., the hollow and solid pentagrams in the FFN structure in the diagram are merged) to further improve hardware efficiency. The merging principle of the sorting operators in FFN is as follows:

[0130] O

[0131]

[0132]

[0133]

[0134] Among them, W down This refers to the weight matrix W of the down module in FFN. up y refers to the weight matrix of the up module in FFN; I refers to the input feature vector (from the output of the MHSA layer structure); f refers to the activation function (in the figure, it is the SiLU function, i.e., Sigmoid-weighted Linear Unit); y refers to the original output of FFN (unsorted); sort() is the sorting operator; O represents the final sorted output. and They represent W respectively up The sorted weights and their relation to W down The weights after sorting.

[0135] As can be seen, by using the above formula, the sorting operation is moved to the weight matrix preprocessing stage, so that only the pre-sorted weights are needed at runtime. and That is, the final output O can be obtained without explicit calculation and sorting, but by first adjusting the weight matrix. as well as Sorting yields the sorted weights. and Then, the FFN calculation is performed directly to obtain the result. That is, sorting the entire output is equivalent to sorting the weights and then performing the FFN structure calculation.

[0136] This concludes the discussion on... Figure 3 The description.

[0137] In this embodiment, the actual application scenario can be any intelligent scenario, such as video surveillance (abnormal behavior detection), customer service dialogue, machine translation, automatic video generation, and other task scenarios. As long as the scenario requires the deployment of a pre-trained model in the edge device, the inference acceleration method for edge devices proposed in this application can be adopted.

[0138] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the inference process for an abnormal behavior monitoring system using conventional methods provided in the embodiments of this application.

[0139] like Figure 4As shown, when the abnormal behavior monitoring system in related technologies receives the sample to be detected, it directly performs inference calculations based on the weights of the large model and outputs the inference results. This method has a large overall computational load and requires a lot of resources.

[0140] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the inference process for the abnormal behavior monitoring system after quantification parameters provided in the embodiments of this application.

[0141] like Figure 5 As shown, the abnormal behavior monitoring system of this application, after quantization of parameters, pre-stores the quantized model parameters and corresponding quantization hyperparameters. When a sample to be detected is received, it can directly quantize the sample according to the stored quantization hyperparameters, further output the detection results, and then perform inverse quantization on the detection results to obtain the final inference results, which greatly reduces the resource consumption.

[0142] Please refer to Figure 6 , Figure 6 This is a structural diagram of a reasoning acceleration device for edge devices proposed in an embodiment of this application. Figure 6 As shown, the device may include a first partitioning unit 601, a second partitioning unit 602, and a quantization processing unit 603. Specifically, the device includes:

[0143] The first partitioning unit 601 is used to divide the weight matrix of the pre-trained model into N weight sub-blocks; where N is greater than 1, and each weight sub-block has a corresponding quantization bit width;

[0144] The second partitioning unit 602 is used to divide the activation matrix of the pre-trained model into M activation sub-blocks; where M is less than or equal to N; each activation sub-block is configured with a corresponding quantization bit width; any activation sub-block corresponds to at least one weight sub-block;

[0145] The quantization processing unit 603 is used to quantize the weight sub-blocks and the corresponding activation sub-blocks in the pre-trained model to obtain the target model. The target model is deployed on an edge device for accelerated inference of the task. When quantizing the weight sub-blocks and the corresponding activation sub-blocks, if at least two weight sub-blocks have the same quantization bit width, the at least two weight sub-blocks are quantized as a whole based on the quantization hyperparameters of the weight values ​​corresponding to the at least two weight sub-blocks and the quantization hyperparameters of the activation values ​​corresponding to the activation sub-blocks of each weight sub-block.

[0146] Optionally, the quantization bit width corresponding to each weighted sub-block is determined according to the following method:

[0147] Determine the reference weight range corresponding to the weight sub-block based on the weight values ​​in the weight sub-block;

[0148] The quantization bit width corresponding to the reference weight range is determined as the quantization bit width corresponding to the weight sub-block;

[0149] And / or, the N weight sub-blocks are of the same size; the activation matrix of the pre-trained model is divided into M activation sub-blocks, including:

[0150] Based on the size of the weight sub-blocks, determine the number M of the weight matrix that is divided along the input channel dimension;

[0151] Based on M, the activation matrix corresponding to the weight matrix is ​​divided into M activation sub-blocks along the input channel dimension;

[0152] And / or, the weight value quantization hyperparameter corresponding to any weighted sub-block is determined according to the following method:

[0153] The quantization interval of the weight value corresponding to the weight sub-block is determined based on the weight value in the weight sub-block and the quantization bit width corresponding to the weight sub-block.

[0154] The weight value quantization interval determines the weight value quantization hyperparameter corresponding to the weight sub-block;

[0155] And / or, the activation value quantization hyperparameter corresponding to any activation sub-block is determined according to the following method:

[0156] The activation value quantization interval corresponding to the activation sub-block is determined based on the activation value in the activation sub-block and the quantization bit width corresponding to the activation sub-block.

[0157] The activation value quantization interval determines the activation value quantization hyperparameter corresponding to the activation sub-block;

[0158] And / or, the pre-trained model includes at least one MHSA layer structure and at least one FFN layer structure, wherein the MHSA layer structure and the FFN layer structure include multiple sorting operators, which are used to sort weight sub-blocks based on quantization bit width; wherein weight sub-blocks with the same quantization bit width are sorted together.

[0159] And / or, the quantization processing unit 603 is also used for:

[0160] If it is found that the first sorting operator in any layer structure is used to sort the first weight matrix and the second sorting operator is used to sort the calculation result of the first weight matrix, then the first sorting operator and the second sorting operator are merged into a whole to sort the first weight matrix and the calculation result of the first weight matrix.

[0161] This concludes the process. Figure 6 Description of inference acceleration devices applied to edge devices.

[0162] This application also provides embodiments that... Figure 6Hardware structure description of the illustrated device. This hardware structure is... Figure 7 The structure in the illustrated electronic device. Please refer to [link / reference]. Figure 7 , Figure 7 This is a structural diagram of an electronic device provided in an embodiment of this application. Figure 7 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.

[0163] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the method disclosed in the above examples of this application.

[0164] For example, the aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For instance, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0165] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for accelerating inference in edge devices, characterized in that, The method includes: The weight matrix of the pre-trained model is divided into N weight sub-blocks; where N is greater than 1, and each weight sub-block has a corresponding quantization bit width. The activation matrix of the pre-trained model is divided into M activation sub-blocks; where M is less than or equal to N; each activation sub-block is configured with a corresponding quantization bit width; each activation sub-block corresponds to at least one weight sub-block; The weight sub-blocks and their corresponding activation sub-blocks in the pre-trained model are quantized to obtain the target model. The target model is deployed on an edge device for accelerated inference of the task. When quantizing the weight sub-blocks and their corresponding activation sub-blocks, if at least two weight sub-blocks have the same quantization bit width, the weight sub-blocks with the same quantization bit width are treated as a whole. The weight sub-blocks with the same quantization bit width are physically stored consecutively, and the activation sub-blocks corresponding to the weight sub-blocks with the same quantization bit width are physically stored consecutively. The at least two weight sub-blocks and their corresponding activation sub-blocks are quantized based on the quantization hyperparameters of the weight values ​​corresponding to the at least two weight sub-blocks and the quantization hyperparameters of the activation values ​​corresponding to the activation sub-blocks of each weight sub-block.

2. The method according to claim 1, characterized in that, The quantization bit width corresponding to each weighted sub-block is determined according to the following method: The reference weight range corresponding to the weight sub-block is determined based on the weight value in the weight sub-block; The quantization bit width corresponding to the reference weight range is determined as the quantization bit width corresponding to the weight sub-block.

3. The method according to claim 1, characterized in that, The N weight sub-blocks are of the same size; the activation matrix of the pre-trained model is divided into M activation sub-blocks, including: Based on the size of the weight sub-blocks, the number M of the weight matrix divided along the input channel dimension is determined; Based on M, the activation matrix corresponding to the weight matrix is ​​divided into M activation sub-blocks along the input channel dimension.

4. The method according to claim 1, characterized in that, The quantization hyperparameter of the weight value corresponding to any weighted sub-block is determined according to the following method: The quantization interval of the weight value corresponding to the weight sub-block is determined based on the weight value in the weight sub-block and the quantization bit width corresponding to the weight sub-block. The weight value quantization interval is used to determine the weight value quantization hyperparameter corresponding to the weight sub-block.

5. The method according to claim 1, characterized in that, The activation value quantization hyperparameter corresponding to any activation sub-block is determined according to the following method: The activation value quantization interval corresponding to the activation sub-block is determined based on the activation value in the activation sub-block and the quantization bit width corresponding to the activation sub-block. The activation value quantization interval is used to determine the activation value quantization hyperparameter corresponding to the activation sub-block.

6. The method according to claim 1, characterized in that, The pre-trained model includes at least one MHSA layer structure and at least one FFN layer structure. The MHSA layer structure and the FFN layer structure include multiple sorting operators, which are used to sort the weight sub-blocks based on the quantization bit width. Among them, weight sub-blocks with the same quantization bit width are sorted together.

7. The method according to claim 6, characterized in that, The method further includes: If it is found that the first sorting operator in any layer structure is used to sort the first weight matrix and the second sorting operator is used to sort the calculation result of the first weight matrix, then the first sorting operator and the second sorting operator are merged into a whole to sort the first weight matrix and the calculation result of the first weight matrix.

8. A reasoning acceleration device for edge devices, characterized in that, The device includes: The first partitioning unit is used to divide the weight matrix of the pre-trained model into N weight sub-blocks; where N is greater than 1, and each weight sub-block has a corresponding quantization bit width; The second partitioning unit is used to divide the activation matrix of the pre-trained model into M activation sub-blocks; where M is less than or equal to N; each activation sub-block is configured with a corresponding quantization bit width; each activation sub-block corresponds to at least one weight sub-block; A quantization processing unit is used to quantize the weight sub-blocks and their corresponding activation sub-blocks in a pre-trained model to obtain a target model. The target model is deployed on an edge device for accelerated inference of the task. When quantizing the weight sub-blocks and their corresponding activation sub-blocks, if at least two weight sub-blocks have the same quantization bit width, the weight sub-blocks with the same quantization bit width are treated as a whole. Weight sub-blocks with the same quantization bit width are physically stored contiguously, and the corresponding activation sub-blocks are also physically stored contiguously. Quantization processing is then performed on the at least two weight sub-blocks and their corresponding activation sub-blocks based on the quantization hyperparameters of the weight values ​​corresponding to the at least two weight sub-blocks and the quantization hyperparameters of the activation values ​​corresponding to the activation sub-blocks of each weight sub-block.

9. The apparatus according to claim 8, characterized in that, The quantization bit width corresponding to each weighted sub-block is determined according to the following method: The reference weight range corresponding to the weight sub-block is determined based on the weight value in the weight sub-block; The quantization bit width corresponding to the reference weight range is determined as the quantization bit width corresponding to the weight sub-block; And / or, the N weighted sub-blocks have the same size; The activation matrix of the pre-trained model is divided into M activation sub-blocks, including: Based on the size of the weight sub-blocks, the number M of the weight matrix divided along the input channel dimension is determined; Based on M, the activation matrix corresponding to the weight matrix is ​​divided into M activation sub-blocks along the input channel dimension; And / or, the weight value quantization hyperparameter corresponding to any weighted sub-block is determined according to the following method: The quantization interval of the weight value corresponding to the weight sub-block is determined based on the weight value in the weight sub-block and the quantization bit width corresponding to the weight sub-block. The weight value quantization interval is used to determine the weight value quantization hyperparameter corresponding to the weight sub-block; And / or, the activation value quantization hyperparameter corresponding to any activation sub-block is determined according to the following method: The activation value quantization interval corresponding to the activation sub-block is determined based on the activation value in the activation sub-block and the quantization bit width corresponding to the activation sub-block. The activation value quantization interval is used to determine the activation value quantization hyperparameter corresponding to the activation sub-block; And / or, the pre-trained model includes at least one MHSA layer structure and at least one FFN layer structure, wherein the MHSA layer structure and the FFN layer structure include multiple sorting operators, which are used to sort the weight sub-blocks based on the quantization bit width; wherein weight sub-blocks with the same quantization bit width are sorted together; And / or, the quantization processing unit is further configured to: If it is found that the first sorting operator in any layer structure is used to sort the first weight matrix and the second sorting operator is used to sort the calculation result of the first weight matrix, then the first sorting operator and the second sorting operator are merged into a whole to sort the first weight matrix and the calculation result of the first weight matrix.

10. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method according to any one of claims 1 to 7.

Images