Post-training quantization method and apparatus, device, and storage medium

Abstract

The present application relates to the technical field of neural networks, and provides a post-training quantization method and apparatus, a device, and a storage medium. The method comprises: acquiring a pre-trained floating-point model; for model parameters in each group of the floating-point model, acquiring model parameters to be adjusted in the group when quantized using a target quantization parameter, wherein the model parameters to be adjusted include top K model parameters in the group, sorted in a descending order of quantization errors; determining corresponding perturbation tensors when the model parameters to be adjusted are quantized, wherein the perturbation tensors are used for adjusting the rounding directions of the model parameters to be adjusted when quantized; and on the basis of the perturbation tensors, using the target quantization parameter to perform quantization processing on the model parameters to be adjusted. By implementing the method of the present application, a perturbation tensor can be introduced for a model parameter having a relatively large quantization error, so as to reduce the quantization error.

Description

Post-training quantization method, device, equipment and storage medium

[0001] This application claims priority to Chinese patent application No. 202311184464.8, filed on September 13, 2023, entitled “Method, device, apparatus and storage medium for post-training quantization”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of neural network technology, and in particular to a method, apparatus, device and storage medium for post-training quantization. Background Art

[0003] Post-training quantization converts floating-point parameters in a model to fixed-point parameters after training. It is a commonly used compression method for deep learning models. During inference, the quantized model can reduce the amount of computation required, effectively improving inference performance.

[0004] In related technologies, when quantizing model parameters in a floating-point model, only the quantization parameters are optimized, which may cause the quantized model to suffer from a large performance loss, thereby affecting the inference performance.

[0005] Summary of the Invention

[0006] The present application provides a method, apparatus, device, and storage medium for post-training quantization, which introduces a perturbation tensor during rounding to improve quantization performance, thereby having a relatively small impact on inference performance.

[0007] In a first aspect, the present application provides a method for post-training quantization, the method comprising: for the model parameters in each group of a floating-point model, obtaining the model parameters to be adjusted in the group when quantized using a target quantization parameter, wherein the target quantization parameter is the quantization parameter corresponding to the group, and the model parameters to be adjusted include the K model parameters in the group that are ranked top in quantization error from large to small, determining the perturbation tensor corresponding to the model parameters to be adjusted during quantization, wherein the perturbation tensor is used to adjust the rounding direction of the model parameters to be adjusted during quantization, and based on the perturbation tensor, using the target quantization parameter to quantize the model parameters corresponding to the perturbation tensor.

[0008] In the solution shown in this application, for the pre-trained floating-point model, after the quantization parameters corresponding to the model parameters in the group are determined, the perturbation tensor is introduced to adjust the rounding direction of the model parameters in the group with relatively large quantization errors, so as to further reduce the quantization error, thereby improving the quantization performance and having a relatively small impact on the inference performance.

[0009] In an optional manner, determining the disturbance tensor corresponding to the model parameter to be adjusted during quantization includes: obtaining a first ratio of the model parameter to be adjusted and a scaling factor included in the target quantization parameter; rounding the sum of the first ratio and the disturbance tensor to obtain a first rounding result, and in the case of asymmetric quantization, determining the sum of the first rounding result and the quantization bias included in the target quantization parameter as the quantized value of the model parameter to be adjusted; and in the case of symmetric quantization, determining the first rounding result as the quantized value of the model parameter to be adjusted; determining the quantization error corresponding to the model parameter to be adjusted based on the quantized value of the model parameter to be adjusted and the value of the model parameter to be adjusted before quantization; and determining the disturbance tensor corresponding to the model parameter to be adjusted when the quantization error corresponding to the model parameter to be adjusted is minimized.

[0010] In the solution shown in the present application, when determining the perturbation tensor, the minimum quantization error is used as a constraint, so that the perturbation tensor with the best quantization performance can be obtained, thereby improving the quantization performance.

[0011] In an optional manner, based on the perturbation tensor, the model parameters corresponding to the perturbation tensor are quantized using the target quantization parameter, including: obtaining a second ratio of the model parameter corresponding to the perturbation tensor and a scaling factor included in the target quantization parameter; rounding the sum of the second ratio and the perturbation tensor to obtain a second rounding result, and in the case of asymmetric quantization, determining the sum of the second rounding result and the quantization bias included in the target quantization parameter as the quantized value of the model parameter corresponding to the perturbation tensor; in the case of symmetric quantization, determining the second rounding result as the quantized value of the model parameter corresponding to the perturbation tensor.

[0012] In an optional manner, the model parameters include input activation values ​​and / or weight values.

[0013] In an optional manner, when the model parameters include weight values, the weight values ​​and scaling factors in the group can be optimized alternately. The process is as follows: the weight values ​​in the group and the scaling factors corresponding to the weight values ​​in the group are optimized alternately using a single variable method to obtain the scaling factors included in the target quantization parameters and the pre-quantized values ​​of the weight values ​​in the group.

[0014] In an optional manner, the method further includes: grouping the weight values ​​in the floating-point model using the channels to which the weight values ​​belong, wherein weight values ​​in the same channel belong to one group.

[0015] In an optional manner, when the model parameters include input activation values, the grouping is performed for the network layers of the floating-point model, and a data distribution of a data set or user input is used to determine a quantization parameter for the input activation value. The processing is as follows: the model parameters include input activation values, the grouping is performed according to the network layers of the floating-point model, and the method further includes: determining the target quantization parameter based on the data distribution of the data set or user input, wherein the distribution conforming to the data set is the same as the distribution conforming to the training data set of the floating-point model, and the data distribution is the distribution conforming to the input activation values ​​of each network layer of the floating-point model.

[0016] In an optional manner, the target quantization parameter is determined based on the data distribution of the user input, including: inputting random data into the floating-point model to obtain the input activation value of the group; performing statistical analysis on the input activation value of the group based on the data distribution to obtain the data range of the input activation value of the group; and determining the target quantization parameter based on the endpoint value of the data range.

[0017] In the solution presented in this application, the user provides the data distribution of input activation values, and using random data as the input of the floating-point model, the quantization parameters of the input activation values ​​are determined and the input activation values ​​of the floating-point model are quantized. Therefore, there is no need to train the data generation model, which can save resources. In addition, there is no need to provide a dataset, which can protect user privacy.

[0018] In a second aspect, the present application provides a post-training quantization apparatus, which has the function of implementing the above-mentioned first aspect. The apparatus includes at least one module, and the at least one module is used to implement the post-training quantization method provided in the above-mentioned first aspect.

[0019] In some embodiments, the modules in the apparatus for post-training quantization are implemented by software, and the modules in the apparatus are program modules. In other embodiments, the modules in the apparatus are implemented by hardware or firmware.

[0020] In a third aspect, the present application provides a computing device comprising a processor and a memory, wherein: the memory stores computer instructions, and the processor executes the computer instructions to implement the method of the first aspect and its optional implementation methods.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium storing computer instructions. When the computer instructions in the computer-readable storage medium are executed by a computing device, the computing device executes the method of the first aspect and its optional implementation method.

[0022] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the method of the above-mentioned first aspect and its optional implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG1 is a schematic diagram of a system architecture provided by an exemplary embodiment of the present application;

[0024] FIG2 is a schematic diagram of a system architecture provided by an exemplary embodiment of the present application;

[0025] FIG3 is a schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present application;

[0026] FIG4 is a flowchart of a method for post-training quantization provided by an exemplary embodiment of the present application;

[0027] FIG5 is a schematic diagram of a process of alternately optimizing quantization parameters of weight values ​​provided by an exemplary embodiment of the present application;

[0028] FIG6 is a schematic diagram of a flow chart of optimizing a rounding operation of a weight value provided by an exemplary embodiment of the present application;

[0029] FIG7 is a schematic structural diagram of a post-training quantization apparatus provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0031] The following is an explanation of some terminology concepts involved in the embodiments of this application.

[0032] 1. Model quantization is the process of converting a floating-point model directly into a fixed-point integer model. For example, a floating-point model with a single-precision floating-point (FP32) or half-precision floating-point (FP16) data type is converted into an integer model with an 8-bit integer (int8) data type. Quantized models can reduce the amount of computation required, effectively improving inference speed and reducing memory usage, enabling the deployment of deep learning models on devices with limited memory.

[0033] 2. Post-training quantization is a commonly used deep learning model compression method. After model training, the floating-point parameters in the model are converted to fixed-point parameters. During the conversion, the quantization parameters used need to be calibrated or optimized to ensure that the accuracy loss after model quantization meets the requirements.

[0034] The following describes the background of the embodiments of the present application.

[0035] Deep learning has a wide range of applications and landing needs in many fields, such as language recognition, image processing, and autonomous driving. Some scenarios, such as autonomous driving, have strict requirements on the latency and memory usage of deep learning model reasoning, so the deep learning model needs to be quantized after training to obtain a deep learning model with low latency and low memory usage. When quantizing the model parameters in the deep learning model, only optimizing the quantization parameters may cause the quantized model to usually have a large performance loss. Based on this, this application introduces a perturbation tensor when quantizing the model parameters to adjust the rounding direction during quantization, so as to improve and reduce the performance loss caused by quantization.

[0036] In the following, the technical solution provided in the embodiment of the present application is described in detail according to the execution subject, system architecture, hardware structure of the device in the system architecture and method flow of the post-training quantization method.

[0037] The following describes the execution of the post-training quantization method.

[0038] The post-training quantization method is performed by a post-training quantization device. Optionally, the device is a hardware device, such as a computing device such as a terminal device or a server. Alternatively, the device is a software device, such as a software program running on the hardware device.

[0039] The following describes the system architecture provided by the embodiments of the present application.

[0040] In an optional manner, an embodiment of the present application provides a system architecture 100. As shown in Figure 1, the system architecture 100 includes a terminal device 101 and a computing device 102. The terminal device 101 and the computing device 102 are connected via a wired or wireless network. The terminal device 101 is a device used by a user, such as a desktop computer, laptop computer, or tablet computer, and is used for the user to interact with the computing device 102. For example, a user can input a pre-trained floating-point model through the terminal device 101. The floating-point model includes multiple network layers connected in sequence. For example, the network layers of the floating-point model may include convolutional layers, fully connected layers, and pooling layers. Different floating-point models may include different network layers. The multiple network layers included in the floating-point model correspond to the application scenarios of the floating-point model. For example, the floating-point model is a deep learning model for text recognition, and the multiple network layers include embedding layers, convolutional layers, and fully connected layers. The computing device 102 can be a server, etc., and the computing device 102 is used to quantize the pre-trained floating-point model and return the quantized model to the terminal device 101.

[0041] In another optional manner, an embodiment of the present application provides another system architecture 200. As shown in Figure 2, the system architecture 200 includes a terminal device 101 and a public cloud 201. The terminal device 101 is connected to the public cloud 201 via a wired or wireless network. The terminal device 101 is a device used by the user, and the terminal device 101 is used for the user to interact with the public cloud 201. The public cloud 201 is an entity that uses basic resources to provide cloud services to users in a cloud computing model. The public cloud 201 can also be considered as a cloud environment. The public cloud 201 includes a cloud data center. The cloud data center includes a large number of basic resources owned by the cloud service provider. The large number of basic resources include computing resources, storage resources and network resources. The computing resources included in the cloud data center can be a computing device cluster. The computing device cluster includes at least one computing device 102. The computing device 102 can be a server, etc. When a user uses a cloud service, the user can input a pre-trained floating-point model to the public cloud 201 through an API or an interactive interface (the interactive interface can be a graphical user interface (GUI)). The computing device in the public cloud 201 receives the pre-trained floating-point model input by the user, quantizes the pre-trained floating-point model, and returns the quantized model to the terminal device 101.

[0042] In the system architecture 200 , the post-training quantization method can be abstracted into a cloud service by the cloud service provider in the public cloud 201 and provided to users. After the user obtains the permission to use the cloud service, he can use the cloud service to perform post-training quantization.

[0043] In another optional embodiment, the present application further provides a system architecture. The system architecture includes a terminal device, which is a device used by a user. The terminal device executes the post-training quantization method of the present application embodiment.

[0044] The following is an introduction to the hardware structure of the device.

[0045] As shown in Figure 3, device 300 is optionally implemented by a general bus architecture and includes at least one processor 301, a communication bus 302, a memory 303, and at least one network interface 304. Device 300 is the terminal device 101 and computing device 102 mentioned above.

[0046] The processor 301 is, for example, a general-purpose central processing unit (CPU), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the solution of the present application. For example, the processor 301 includes an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0047] Communication bus 302 is used to transmit information between the above components. Communication bus 302 can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 3, but this does not mean that there is only one bus or one type of bus.

[0048] The memory 303 is, for example, a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 303 is, for example, independent and connected to the processor 301 via the communication bus 302. The memory 303 can also be integrated with the processor 301.

[0049] Optionally, the memory 303 is used to store a pre-trained floating-point model. When the device 300 needs to use the floating-point model, the processor 301 accesses the memory 303 to obtain the floating-point model stored in the memory 303.

[0050] The network interface 304 uses any transceiver-like device for communicating with other devices or communication networks. The network interface 304 includes a wired network interface and may also include a wireless network interface. For example, the wired network interface may be an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless network interface may be a wireless local area network (WLAN) interface, a cellular network interface, or a combination thereof.

[0051] In a specific implementation, as an example, the processor 301 may include one or more CPUs.

[0052] In a specific implementation, as an example, the device 300 may include multiple processors. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0053] In some embodiments, the memory 303 is used to store the program code 3031 for executing the post-training quantization in this application, and the processor 301 executes the program code 3031 stored in the memory 303. That is, the device 300 can implement the post-training quantization method provided in the method embodiment through the processor 301 and the program code 3031 in the memory 303.

[0054] The following describes the post-training quantization method flow with reference to FIG4 .

[0055] In post-training quantization, the quantized value of the weight can be expressed as W q =W+ΔW, where W represents the value of the weight value before quantization and ΔW represents the difference between the weight value before and after quantization. The value of the input activation value after quantization is represented as A q =A+ΔA, where A represents the value of the input activation value before quantization, ΔA represents the difference between the input activation value before and after quantization, and the final calculated value after quantization is expressed as formula (1). q A q =WA+AΔW+WΔA+ΔWΔA (1)

[0056] The goal of quantization is to minimize the quantization error, which is expressed as formula (2).

[0057] In the embodiment of the present application, the quantization error is optimized. Combining formula (1) and formula (2), it can be seen that the quantization error mainly comes from the quantization of the input activation value and the quantization of the weight value. Therefore, in the embodiment of the present application, the input activation value quantization and the weight value quantization are performed separately to minimize the quantization error. In addition, the quantization process can also be performed with the goal of the quantization error being less than or equal to a certain value.

[0058] As can be seen from the above description, post-training quantization includes two parts: input activation value quantization and weight value quantization, see Figure 4. The quantization of the input activation value includes quantization based on rounding optimization and quantization based on quantization parameters. When the input activation value is quantized, the input of post-training quantization includes a pre-trained floating-point model and data distribution. The data distribution is the distribution statistics of the input activation value. For example, the data distribution is a normal distribution, etc. Alternatively, when the input activation value is quantized, the input of post-training quantization includes a pre-trained floating-point model and a data set. The data distribution of the data set is the same as the data distribution of the training data set of the training floating-point model. The pre-trained floating-point model is a deep learning model that has been trained. The data type of the parameters in the deep learning model is floating-point. The deep learning model includes deep learning models for scenarios such as speech recognition, text recognition, image processing, or vehicle driving. The embodiments of this application are not limited. The data type in the pre-trained floating-point model is any floating-point type, for example, FP32, FP16, or FP64. When the weight value is quantized, the input of post-training quantization includes a pre-trained floating-point model or a floating-point model after the input activation value is quantized. Weight value quantization includes weight value quantization based on an alternating method and weight value quantization based on rounding optimization. There is no order in which the input activation value quantization and the weight value quantization are performed. The input activation value can be quantized first, or the weight value can be quantized first. This is not limited in the embodiment of the present application. Figure 4 shows that the input activation value is quantized first, and then the weight value is quantized. After the input activation value quantization and the weight value quantization, the quantized model is output.

[0059] In the embodiment of the present application, when quantization is performed based on rounding optimization, the model parameters of the floating-point model include input activation values ​​and / or weight values. The following describes the process of input weight value quantization and the process of input activation value quantization respectively.

[0060] 1. The process of weight quantization: The process of weight quantization includes the process of determining the quantization parameters of the weight and the process of optimizing the rounding operation.

[0061] Before describing the weight quantization process, we first group the weights in the floating-point model. Two possible methods are provided below.

[0062] Method 1 groups the weights in a floating-point model according to the channel to which they belong. Weights belonging to the same channel are grouped together. For example, the weights of a convolutional layer in a floating-point model are four-dimensional tensors represented as s×e×h×r, where s represents the number of output channels, e represents the number of input channels, and h×r represents the h×r convolution kernel. During quantization, e×h×r weights form a group.

[0063] Method 2: Group the weight values ​​in the floating-point model according to the network layer to which the weight values ​​belong. The weight values ​​belonging to the same network layer are grouped into the same group. In this way, the number of network layers in the floating-point model is equal to the number of weight value groups.

[0064] 1.1. The process of optimizing the quantization parameters of the weight values ​​and obtaining the target quantization parameters includes alternately optimizing the scaling factors of the weight values ​​and optimizing the scaling factors and quantization bias of the weight values.

[0065] 1.1.1. In the case of symmetric quantization, the scaling factors and weight values ​​of the weight values ​​are alternately optimized to obtain the target quantization parameter, as shown in steps 501 to 502 of the process shown in FIG5 . For each weight value in each group of the floating-point model, the quantization parameter corresponding to each weight value in the group is the same. In other words, the target quantization parameter corresponding to the group is determined through the process shown in FIG5 .

[0066] Step 501: Obtain a pre-trained floating-point model.

[0067] In this embodiment, the user inputs the pre-trained floating-point model through the terminal device, and the computing device receives the pre-trained floating-point model sent by the terminal device, or the computing device obtains the stored pre-trained floating-point model.

[0068] Step 502: Optimize the quantization parameters of the weight values ​​using an alternating optimization method.

[0069] In this embodiment, when alternately optimizing the quantization parameters of the weight values, a symmetric quantization method is considered, and the quantization parameters include a scaling factor. For the weight values ​​in each group of the floating-point model, the scaling factor and the weight value are separated, and the scaling factor and the weight value are optimized separately using the alternating optimization method to reduce the amount of calculation. Specifically, in step one, the weight value is fixed, and the scaling factor is optimized according to formula (3) to minimize the quantization error when the weight value is fixed. In step two, the scaling factor is fixed, and the weight value is optimized according to formula (4) to minimize the quantization error when the scaling factor is fixed. Step one and step two are further repeated until the quantization error is minimized, or when the quantization error is less than or equal to the target value, the corresponding scaling factor and the weight value in the group before quantization are obtained. The target value can be set according to actual needs, or after steps one and two are repeated M times, the last scaling factor and the weight value in the group are obtained. M can be set according to actual needs. Subsequently, during quantization, the optimized weight value is the value before quantization of the weight value. For each group, the scaling factor can be determined using this method.

[0070] Where L(W1, W2) represents the quantization error of the weight value in a group. The expression is shown in formula (5) or (6). In formula (3), Indicates the value of scale1 when minimizing the quantization error. Scale1 represents the scaling factor. In formula (4), When minimizing the quantization error, the value of W1 and W2 are determined using formula (7).

[0071] In formula (5), W1 represents the value of the weight value before quantization, W2 represents the value of the weight value after quantization and the value after dequantization, W 1j Represents the value of the jth weight value before quantization, j is an integer less than or equal to g, W 2j represents the dequantized value of the jth weight value after quantization, and g represents the number of weight values ​​in the group. In formula (7), round represents a rounding operation, a floor operation, or an upward rounding operation.

[0072] It should be noted that the process shown in Figure 5 is for symmetric quantization. In the case of asymmetric quantization, the above method can also be used, except that the quantization bias needs to be determined. In addition, after the pre-quantization value of the weight value is obtained in Figure 5, the pre-quantization value of the weight value obtained in Figure 5 is used in subsequent quantization.

[0073] 1.1.2. The process of determining the target quantization parameter of the weight value.

[0074] For the weight values ​​in each group of the floating-point model, the quantization parameters corresponding to the weight values ​​in the group are the same, so they can be described as the target quantization parameters corresponding to the group. In the case of asymmetric quantization, the target quantization parameters corresponding to the group are determined according to formula (8), and in the case of symmetric quantization, the target quantization parameters corresponding to the group are determined according to formula (9).

[0075] Where L(W1, W2) represents the quantization error of the weight value in a group. The expression is shown in formula (5) or (6). In formula (8), Indicates the values ​​of scale1 and offset1 when minimizing the quantization error. Offset1 represents the quantization bias. W2 is determined using formula (10). In formula (9), When minimizing the quantization error, the value of scale1 and W2 are determined using formula (7).

[0076] In formula (8), two parameters, the scaling factor scale1 and the quantization offset offset1, need to be adjusted. When adjusting these two parameters, the initial values ​​of the two parameters are preset values. Under the premise of fixing scale1, adjust offset1 so that the quantization error L(W1, W2) of the weight value in the group is minimized. Then, fix offset1 to the offset1 when L(W1, W2) is minimized, and adjust scale1 so that L(W1, W2) is minimized. This process is performed until the number of times reaches the preset number, or the minimum quantization error is obtained, or the quantization error obtained is less than or equal to the target value, and the final scale1 and offset1 are obtained. The preset number is set according to an empirical value, such as 15.

[0077] It should be noted that both 1.1.1 and 1.1.2 can obtain the target quantization parameter, and are two ways to determine the target quantization parameter.

[0078] 1.2. Optimizing the rounding operation process: See steps 601 to 603 in the process shown in FIG6 . After obtaining the target quantization parameter, execute the process shown in FIG6 .

[0079] Step 601: For each group of floating-point model model parameters, obtain the model parameters to be adjusted in the group when quantized using the target quantization parameter, wherein the model parameters to be adjusted include the top K model parameters in the group whose quantization errors are ranked from large to small.

[0080] The rounding operation during quantization is a rounding operation, a rounding down operation, a rounding up operation, or the like.

[0081] In this embodiment, for the weight values ​​in each group of the floating-point model, when no perturbation tensor is introduced, in the case of symmetric quantization, formula (7) is used to quantize and round the weight values ​​in the group using the target quantization parameter to obtain the quantized value of the weight value. In the case of asymmetric quantization, formula (10) is used to quantize and round the weight values ​​in the group using the target quantization parameter to obtain the quantized value of the weight value. Formula (5) or formula (6) is used to calculate the quantization error of the weight value in the group. The quantization errors corresponding to the weight values ​​in the group are arranged in order from large to small to determine the top K weight values. The value of K is set according to the number of weight values ​​in the group. The more weight values ​​in the group, the larger the value of K, and the fewer weight values ​​in the group, the smaller the value of K. The K weight values ​​are considered to have relatively large quantization errors and are called weight values ​​to be adjusted.

[0082] Step 602: Determine a disturbance tensor corresponding to the to-be-adjusted model parameter during quantization, wherein the disturbance tensor is used to adjust a rounding direction during quantization of the to-be-adjusted model parameter.

[0083] In this embodiment, an initial perturbation tensor is introduced for the weight value to be adjusted. In the case of symmetric quantization, the weight value to be adjusted is quantized using formula (11) to obtain the quantized value of the weight value. In the case of asymmetric quantization, the weight value to be adjusted is quantized using formula (12) to obtain the quantized value of the weight value. Formula (5) or (6) is used to calculate the quantization error of the weight value to be adjusted in the group (in this case, g is the number of weight values ​​to be adjusted), and it is determined whether the quantization error is minimum. If not, the perturbation tensor is updated until the quantization error is minimum, and the perturbation tensor used when the quantization error is minimum is obtained. If it is minimum, the perturbation tensor used when the quantization error is minimum is obtained. The perturbation tensor is the first perturbation tensor. The process is represented by formula (13). Alternatively, the quantization error of each weight value to be adjusted is calculated to determine whether the quantization error of each weight value to be adjusted is minimum. If not, the perturbation tensor is updated until the quantization error is minimum, and the perturbation tensor is obtained. If it is minimum, the perturbation tensor is obtained.

[0084] Wherein, in formula (11) and formula (12), Δ1 represents the first disturbance tensor introduced. In formula (13), The value of Δ1 when minimizing the quantization error, and the expression of L(W1, W2) can be found in formula (5) or (6).

[0085] It should be noted that the principle of introducing the perturbation tensor is: when using the rounding operation, the decimal part of the original ratio of the weight value to the scaling factor is less than 5. By introducing the perturbation tensor, the decimal part is changed to greater than or equal to 5, thereby changing the rounded value, or the decimal part of the original ratio of the weight value to the scaling factor is greater than or equal to 5. By introducing the perturbation tensor, the decimal part is changed to less than 5, thereby changing the rounded value. When using the rounding operation, the integer part of the original ratio of the weight value to the scaling factor is x, which is rounded up to x+1. The perturbation tensor is introduced, and the perturbation tensor is added to the ratio of the weight value to the scaling factor (the perturbation tensor is a negative number at this time), and the integer part becomes x-1, which is rounded up to x, achieving a change in the rounding direction. Other situations are similar and will not be repeated here.

[0086] It should also be noted that the first disturbance tensors corresponding to the K weight values ​​included in the weight values ​​to be adjusted may be the same, partially the same, or completely different.

[0087] Step 603: Based on the disturbance tensor, the model parameters corresponding to the disturbance tensor are quantized using the target quantization parameter.

[0088] In this embodiment, in the case of symmetric quantization, the computing device uses formula (11) to quantize the weight value to be adjusted. In the case of asymmetric quantization, the computing device uses formula (12) to quantize the weight value to be adjusted.

[0089] By adopting the process shown in FIG6 , a perturbation tensor is introduced for the weight values ​​with large quantization errors in the group, which can further reduce the quantization error.

[0090] It should be noted that when 1.1 and 1.2 exist at the same time, it is equivalent to executing the content in 1.1 when the perturbation tensor is 0 to determine the target quantization parameter, and then executing the content in 1.2 to adjust the rounding direction when the determined target quantization parameter is fixed.

[0091] In addition, in the process of Figure 6, it is to determine whether the quantization error of the weight value to be adjusted is the smallest, or it can be to determine whether the quantization error of all weight values ​​in the group is less than a certain value. When the quantization error is less than or equal to the value, the perturbation tensor is determined, and the value is smaller than the target value used when determining the target quantization parameter.

[0092] 2. Quantization of input activation values: This quantization process includes determining the target quantization parameters and optimizing the rounding operation.

[0093] 2.1. The process of determining the target quantitative parameters. This determination process includes two methods: method 1 is a determination method with data, and method 2 is a determination method without data.

[0094] Method 1: Method for determining data, see steps 701 to 704.

[0095] Step 701: Obtain a data set input by a user.

[0096] The data set includes multiple data, and the distribution of the data in the data set is the same as the distribution of the data in the training data set. The training data set is a data set used to obtain a floating-point model.

[0097] In this embodiment, the user inputs a data set through a terminal device, and the computing device receives the data set sent by the terminal device.

[0098] Step 702: Input the data in the data set into the floating-point model to obtain input activation values ​​of each network layer of the floating-point model.

[0099] In this embodiment, the computing device inputs the data from the dataset into a floating-point model and performs forward inference to obtain input activation values ​​for each network layer of the floating-point model. For the first network layer among the multiple network layers sequentially connected in the floating-point model, the input activation value of this network layer can be the features of the data from the dataset. For any network layer other than the first network layer among the multiple network layers, the input activation value of this network layer is the feature data output by the previous network layer located before it.

[0100] Step 703: Perform statistical analysis on the input activation values ​​of each network layer to obtain the data range of the input activation values ​​of each network layer.

[0101] In this embodiment, for any network layer, the computing device performs statistical analysis on the input activation values ​​of the network layer, determines the data range of the majority of the input activation values, and determines this data range as the data range of the input activation values ​​of the network layer. For example, the data range containing 99% of the input activation values ​​is determined as the data range of the input activation values ​​of the network layer.

[0102] Step 704: Determine the quantization parameters of the input activation values ​​of each network layer based on the endpoint values ​​of the data range corresponding to each network layer.

[0103] In this embodiment, in the case of asymmetric quantization, the target quantization parameter includes a scaling factor and a quantization bias. For any network layer, the computing device uses formula (14) to calculate the initial value of the scaling factor corresponding to the network layer, and uses formula (15) to calculate the initial value of the quantization bias corresponding to the network layer.

[0104] In formulas (14) and (15), scale2 represents a scaling factor, max_val represents the maximum value of the input activation value of the network layer, min_val represents the minimum value of the input activation value of the network layer, n represents the number of quantization bits, offset2 represents the quantization bias, and round represents a rounding operation, a rounding-down operation, or a rounding-up operation.

[0105] Then, the computing device uses the initial value of the quantization parameter and uses formula (16) to quantize the input activation value of the network layer to obtain the quantized value of the input activation value, dequantizes the quantized value of the input activation value to obtain the dequantized value, and uses formula (17) or (18) to calculate the quantization error of the input activation value of the network layer. It is judged whether the quantization error is less than or equal to the first value corresponding to the network layer. If the quantization error is less than or equal to the first value, the initial value of the scaling factor is determined as the value of the scaling factor, and the initial value of the quantization bias is determined as the value of the quantization bias. If the quantization error is greater than the first value, the initial value of the quantization parameter of the input activation value of the network layer is updated, and the above process is continued until the quantization error of the input activation value of the network layer is less than or equal to the first value, and the target quantization parameter of the input activation value of the network layer is obtained. Alternatively, the quantization error of the input activation value of the network layer is calculated using formula (17) or (18). Update the initial value of the quantization parameter of the input activation value of the network layer, and continue the above process until the quantization error of the input activation value of the network layer is minimized, thereby obtaining the target quantization parameter of the input activation value of the network layer. The first value can be set based on actual conditions. The more the number of input activation values ​​of the network layer, the larger the value of K, and the fewer the number of input activation values ​​of the network layer, the smaller the value of K.

[0106] In formula (16), A2 represents the value of the input activation value after quantization, A1 represents the value of the input activation value before quantization, scale2 represents the scaling factor, offset2 represents the quantization bias, and round represents the rounding operation, or the rounding down operation, or the rounding up operation.

[0107] In formulas (17) and (18), L(A1, A2) represents the quantization error, A1i represents the value of the i-th input activation value before quantization, i is an integer less than or equal to m, and A 2i It represents the dequantized value of the quantized value of the i-th input activation value. The dequantized value is a floating point number. m represents the number of input activation values ​​of the network layer.

[0108] Optionally, in the case of asymmetric quantization, the computing device may use an alternating update method to determine the scaling factor and the quantization bias. For example, for any network layer, when the scaling factor is an initial value, the initial value of the scaling factor and the initial value of the quantization bias are used to quantize the input activation value of the network layer to obtain the quantized value of the input activation value of the network layer, see formula (16).

[0109] The computing device then dequantizes the quantized value and uses formula (17) or (18) to calculate the quantization error corresponding to the network layer. The computing device increases the initial value of the quantization bias by a preset adjustment amplitude, substitutes the initial value of the scaling factor and the adjusted quantization bias into formula (16), quantizes the input activation value of the network layer, obtains the quantized value of the input activation value of the network layer, and uses formula (17) or (18) to calculate the quantization error corresponding to the network layer, and determines whether the quantization error is minimum. If not, continue to adjust the quantization bias until the value of the quantization bias is obtained when the quantization error is minimum.

[0110] Then, when the quantization bias is obtained, the quantization bias is fixed and the scaling factor is updated to determine the value of the scaling factor when the quantization error is minimized. The process of updating the scaling factor is similar to the process of updating the quantization bias and will not be repeated here.

[0111] In this way, after multiple alternating updates of the quantization bias and the scaling factor, the final scaling factor and quantization bias are obtained.

[0112] In the case of symmetric quantization, the target quantization parameter includes a scaling factor. For any network layer, the computing device uses formula (14) to calculate the initial value of the scaling factor corresponding to the network layer. For any network layer, only the scaling factor corresponding to the network layer is adjusted, which is equivalent to setting the quantization bias to 0. The process of determining the scaling factor refers to the process of asymmetric quantization and is not repeated here.

[0113] Method 2: Method for determining if there is no data, see steps 801 to 804.

[0114] Step 801: Obtain data distribution input by the user.

[0115] In this embodiment, the user inputs data distribution through the terminal device, and the computing device receives the data distribution sent by the terminal device.

[0116] Step 802: input random data into the floating-point model to obtain input activation values ​​of each network layer of the floating-point model.

[0117] In this embodiment, the computing device obtains stored random data, which causes the input activation value to conform to the data distribution. Alternatively, the computing device generates random data conforming to the data distribution based on the data distribution. Alternatively, the computing device stores random data corresponding to the data distribution, and the computing device obtains random data conforming to the data distribution input by the user from the stored random data. The computing device inputs the random data into a floating-point model and performs forward reasoning to obtain input activation values ​​for each network layer of the floating-point model. For the network layer that is the first of multiple network layers connected sequentially in the floating-point model, the input activation value of the network layer can be a feature of the random data. For any network layer other than the first of the multiple network layers, the input activation value of the network layer is the feature data output by the previous network layer located before the network layer.

[0118] Step 803: Based on the data distribution, statistical analysis is performed on the input activation values ​​of each network layer to obtain the data range of the input activation values ​​of each network layer.

[0119] In this embodiment, the data distribution of the user input is the data distribution of the input activation values ​​of each network layer. For any network layer, the computing device uses this data distribution to determine the maximum and minimum values ​​of the input activation values ​​of the network layer, and obtains the data range of the input activation values. For example, if the data distribution is a normal distribution, for the kth network layer of the floating-point model, the input activation values ​​of the kth network layer can be used to calculate the parameters μ and σ of the normal distribution, where μ is the mathematical expectation and σ is the standard deviation. In the normal distribution, the probability that the data falls within (μ-3σ, μ+3σ) is 99.74%, so (μ-3σ, μ+3σ) can be used as the data range of the input activation values ​​of the kth network layer, where μ-3σ is the minimum value of the data range and μ+3σ is the maximum value of the data range.

[0120] Step 804: Determine target quantization parameters of the input activation values ​​of each network layer based on the endpoint values ​​of the data range.

[0121] The description of the processing procedure of step 804 refers to the description of the processing procedure of step 704 and is not repeated here.

[0122] As can be seen from Method 2, when performing model quantization, users do not need to provide data, which protects user privacy and has a wider range of applications. Furthermore, no additional training is required to obtain the data generator, which saves resources and improves quantization efficiency. Furthermore, the quantization parameters of the input activation values ​​are initialized based on their data distribution, resulting in higher accuracy than random estimation. Furthermore, there are no requirements for the structure of the floating-point model, which enhances generalization capabilities.

[0123] After obtaining the quantization parameters of the input activation values ​​in steps 704 and 804, the computing device returns the quantization parameters to the terminal device, which stores the quantization parameters. Subsequently, in the inference process, the quantization parameters are used to quantize the input activation values, which can reduce the amount of calculation of the input activation values.

[0124] 2.2. After determining the target quantization parameter of the input activation value, optimize the rounding operation during the quantization process of the input activation value. For the optimization process, see steps 901 to 903.

[0125] Step 901: For the input activation values ​​in each network layer of the floating-point model, obtain the input activation values ​​to be adjusted in the group when quantized using the target quantization parameter, wherein the input activation values ​​to be adjusted include the K input activation values ​​in the group that are ranked top in descending order of quantization error.

[0126] In this embodiment, for the input activation values ​​in each network layer of the floating-point model, when no perturbation tensor is introduced, in the case of symmetric quantization, formula (19) (the quantization bias is 0) is used to quantize and round the input activation values ​​in the network layer to obtain the quantized value of the input activation value. In the case of asymmetric quantization, formula (20) is used to quantize and round the input activation values ​​in the network layer to obtain the quantized value of the input activation value. Formula (17) or formula (18) is used to calculate the quantization error of the input activation value in the network layer. The quantization errors corresponding to the input activation values ​​in the network layer are arranged in order from large to small to determine the top K input activation values. The value of K is set according to the number of input activation values ​​in the network layer. The more the number of input activation values ​​in the network layer, the larger the value of K, and the fewer the number of input activation values ​​in the network layer, the smaller the value of K. These K input activation values ​​are considered to have relatively large quantization errors and are called input activation values ​​to be adjusted.

[0127] Step 902: Determine a second perturbation tensor corresponding to the input activation value to be adjusted during quantization and rounding, wherein the second perturbation tensor is used to adjust the rounding direction of the input activation value to be adjusted during quantization.

[0128] In this embodiment, an initial perturbation tensor is introduced for the input activation value to be adjusted. In the case of symmetric quantization, the input activation value to be adjusted is quantized and rounded using formula (19) to obtain the quantized value of the input activation value. In the case of asymmetric quantization, the input activation value to be adjusted is quantized and rounded using formula (20) to obtain the quantized value of the input activation value. The quantization error of the input activation value to be adjusted is calculated using formula (17) or (18) (in this case, m is the number of input activation values ​​to be adjusted), and it is determined whether the quantization error is minimized. If not, the perturbation tensor is updated until the quantization error is minimized, and the perturbation tensor used when the quantization error is minimized is obtained. If the quantization error is minimized, the perturbation tensor used when the quantization error is minimized is obtained. This perturbation tensor is the second perturbation tensor. This process is represented by formula (21). Alternatively, the quantization error of each input activation value to be adjusted is calculated to determine whether the quantization error of each input activation value to be adjusted is minimized. If not, the perturbation tensor is updated until the quantization error is minimized, and the perturbation tensor is obtained. If it is minimized, the perturbation tensor is obtained.

[0129] Wherein, in formula (19) and formula (20), Δ2 represents the second perturbation tensor introduced. In formula (21), The value of Δ2 when minimizing the quantization error, and the expression of L(A1, A2) can be found in formula (17) or (18).

[0130] It should be noted that the second perturbation tensors corresponding to the K input activation values ​​included in the input activation values ​​to be adjusted may be the same, partially different, or completely different.

[0131] Step 903: During the inference process, based on the second perturbation tensor, the input activation value corresponding to the second perturbation tensor is quantized using the target quantization parameter.

[0132] In this embodiment, since input data is input into the quantized model, there will be input activation values. Therefore, step 903 is a process performed during the inference process. In the inference process using the quantized model, in the case of symmetric quantization, the computing device uses formula (19) to quantize the input activation values ​​corresponding to the second perturbation tensor, and also uses formula (19) to quantize other input activation values ​​that do not correspond to the second perturbation tensor, except that the second perturbation tensor takes a value of 0. In the case of asymmetric quantization, the computing device uses formula (20) to quantize the input activation values, and also uses formula (20) to quantize other input activation values ​​that do not correspond to the second perturbation tensor, except that the second perturbation tensor takes a value of 0.

[0133] It should be noted that the weight value quantization and the input activation value quantization can both adopt an asymmetric quantization method, or both adopt a symmetric quantization method, or one can adopt a symmetric quantization method and the other an asymmetric quantization method.

[0134] When describing the quantization of input activation values, they are grouped by network layer. Of course, other grouping methods can also be used, for example, grouping input activation values ​​by channel.

[0135] In the embodiments of the present application, the post-training quantization method does not rely on the structure of the floating-point model, has higher generalization, and can be widely applied to typical convolutional neural network (CNN) models or (converter) Transformer models. In addition, the embodiments of the present application also optimize the rounding direction, which can improve the accuracy of weight value quantization and input activation value quantization, and improve the inference accuracy of the quantized model.

[0136] In the embodiments of this application, products that may be involved include model compression tools or inference model deployment tools. Subsequent infringement determinations can be made using the following methods:

[0137] 1. From the process and log analysis of model compression, whether the quantization parameters of the weight value are alternately optimized during the quantization process, and whether the rounding direction when quantizing the weight value using the quantization parameters of the weight value is the same as the rounding direction of the embodiment of the present application.

[0138] 2. Check the functions released by the model compression tool to see whether the above-mentioned post-training quantization process is integrated.

[0139] FIG7 is a block diagram of a post-training quantization apparatus provided in an embodiment of the present application. The apparatus can be implemented as part or all of the apparatus through software, hardware, or a combination of both. The apparatus provided in an embodiment of the present application can implement the processes described in FIG4 to FIG6 of the embodiment of the present application, and the apparatus includes: an acquisition module 710, a determination module 720, and a quantization module 730, wherein:

[0140] An acquisition module 710 is configured to acquire, for each group of floating-point models, model parameters to be adjusted in the group when quantized using a target quantization parameter, wherein the target quantization parameter is the quantization parameter corresponding to the group, and the model parameters to be adjusted include K model parameters in the group that are ranked top in descending order of quantization error. Specifically, the acquisition module 710 can be used to implement the acquisition function of step 601 and execute the implicit steps included in step 601.

[0141] A determination module 720 is configured to determine a perturbation tensor corresponding to the to-be-adjusted model parameter during quantization, wherein the perturbation tensor is used to adjust a rounding direction during quantization of the to-be-adjusted model parameter. Specifically, the determination module 720 may be configured to implement the determination function of step 602 and to execute the implicit steps included in step 602.

[0142] The quantization module 730 is used to quantize the model parameters corresponding to the perturbation tensor using the target quantization parameter based on the perturbation tensor, and can be specifically used to implement the quantization function of step 603 and execute the implicit steps included in step 603.

[0143] In an optional manner, the determining module 720 is configured to:

[0144] Obtaining a first ratio of the model parameter to be adjusted to a scaling factor included in the target quantization parameter;

[0145] performing rounding processing on a sum of the first ratio and the disturbance tensor to obtain a first rounding result; in the case of asymmetric quantization, determining the sum of the first rounding result and a quantization bias included in the target quantization parameter as the quantized value of the to-be-adjusted model parameter; and in the case of symmetric quantization, determining the first rounding result as the quantized value of the to-be-adjusted model parameter;

[0146] Determining a quantization error corresponding to the model parameter to be adjusted based on a quantized value of the model parameter to be adjusted and a value of the model parameter to be adjusted before quantization;

[0147] Determine a disturbance tensor corresponding to the to-be-adjusted model parameter when the quantization error corresponding to the to-be-adjusted model parameter is minimized.

[0148] In an optional manner, the quantization module 730 is configured to:

[0149] Obtaining a second ratio of a model parameter corresponding to the disturbance tensor to a scaling factor included in the target quantization parameter;

[0150] The second ratio and the sum of the perturbation tensor are rounded to obtain a second rounded result. In the case of asymmetric quantization, the sum of the second rounded result and the quantization bias included in the target quantization parameter is determined as the quantized value of the model parameter corresponding to the perturbation tensor. In the case of symmetric quantization, the second rounded result is determined as the quantized value of the model parameter corresponding to the perturbation tensor.

[0151] In an optional manner, the model parameters include input activation values ​​or weight values.

[0152] In an optional manner, the model parameters include weight values, and the determining module 720 is further configured to:

[0153] The weight values ​​in the group and the scaling factors corresponding to the weight values ​​in the group are optimized alternately using a single variable method to obtain the scaling factors included in the target quantization parameters and the values ​​of the weight values ​​in the group before quantization.

[0154] In an optional manner, the model parameters include input activation values, the grouping is performed according to the network layer of the floating-point model, and the determining module 720 is further configured to:

[0155] The target quantization parameter is determined based on a data distribution of a data set or user input, wherein the distribution conforming to the data set is the same as the distribution conforming to the training data set of the floating-point model, and the data distribution is the distribution conforming to the input activation values ​​of each network layer of the floating-point model.

[0156] In an optional manner, the determining module 720 is further configured to:

[0157] Inputting random data into the floating point model to obtain input activation values ​​of the group;

[0158] Based on the data distribution, performing statistical analysis on the input activation values ​​of the group to obtain a data range of the input activation values ​​of the group;

[0159] The target quantization parameter is determined based on the endpoint values ​​of the data range.

[0160] The detailed process of weight value quantization performed by the apparatus for post-training quantization shown in FIG7 is referred to the description in the previous embodiments and will not be repeated here. The apparatus for post-training quantization shown in FIG7 is the computing device 102 in FIG1 and FIG2.

[0161] In an optional embodiment, a computer program product is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computing device to execute the processes shown in FIG4 to FIG6.

[0162] Those skilled in the art will appreciate that the various method steps and units described in the embodiments disclosed in this application can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0163] In the several embodiments provided in this application, it should be understood that the disclosed system architecture, device and method can be implemented in other ways. For example, the device embodiment described above is only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.

[0164] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0165] In addition, the modules in the various embodiments of the present application may be integrated into a single processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The integrated modules may be implemented in the form of hardware or software modules.

[0166] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0167] The term "and / or" in this application includes three cases, for example, A and / or B includes three cases of A, B, and A and B. The term "plurality" in this application means two or more.

[0168] The above description is merely an exemplary embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A post-training quantization method, characterized in that: The method comprises: For the model parameters in each group of the floating-point model, obtain the model parameters to be adjusted in the group when quantized using the target quantization parameter, wherein the target quantization parameter is the quantization parameter corresponding to the group, and the model parameters to be adjusted include K model parameters in the group that are ranked top in descending order of quantization error; Determine a disturbance tensor corresponding to the to-be-adjusted model parameter when quantized, wherein the disturbance tensor is used to adjust a rounding direction when quantizing the to-be-adjusted model parameter; Based on the disturbance tensor, the model parameters corresponding to the disturbance tensor are quantized using the target quantization parameter.

2. The method according to claim 1, characterized in that The determining of the disturbance tensor corresponding to the to-be-adjusted model parameter during quantization includes: Acquire a first ratio of the model parameter to be adjusted to a scaling factor included in the target quantization parameter; performing rounding processing on the sum of the first ratio and the disturbance tensor to obtain a first rounding result, and in the case of asymmetric quantization, determining the sum of the first rounding result and the quantization bias included in the target quantization parameter as the quantized value of the model parameter to be adjusted; and in the case of symmetric quantization, determining the first rounding result as the quantized value of the model parameter to be adjusted; Determining a quantization error corresponding to the model parameter to be adjusted based on a quantized value of the model parameter to be adjusted and a value of the model parameter to be adjusted before quantization; Determine a disturbance tensor corresponding to the to-be-adjusted model parameter when the quantization error corresponding to the to-be-adjusted model parameter is minimum.

3. The method according to claim 1 or 2, characterized in that: The step of quantizing the model parameters corresponding to the disturbance tensor using the target quantization parameter based on the disturbance tensor includes: Obtaining a second ratio of a model parameter corresponding to the disturbance tensor to a scaling factor included in the target quantization parameter; The second ratio and the sum of the perturbation tensor are rounded to obtain a second rounded result. In the case of asymmetric quantization, the sum of the second rounded result and the quantization bias included in the target quantization parameter is determined as the quantized value of the model parameter corresponding to the perturbation tensor. In the case of symmetric quantization, the second rounded result is determined as the quantized value of the model parameter corresponding to the perturbation tensor.

4. The method according to any one of claims 1 to 3, characterized in that: The model parameters include input activation values ​​or weight values.

5. The method according to any one of claims 1 to 3, characterized in that: The model parameters include weight values, and the method further includes: The weight values ​​in the group and the scaling factors corresponding to the weight values ​​in the group are optimized alternately by a single variable method to obtain the scaling factors included in the target quantization parameters and the values ​​of the weight values ​​in the group before quantization.

6. The method according to any one of claims 1 to 3, characterized in that: The model parameters include input activation values, the grouping is grouped according to the network layer of the floating point model, and the method further includes: The target quantization parameter is determined based on a data set or a data distribution input by a user, wherein the distribution conforming to the data set is the same as the distribution conforming to the training data set of the floating-point model, and the data distribution is the distribution conforming to the input activation values ​​of each network layer of the floating-point model.

7. The method according to claim 6, characterized in that Determining the target quantization parameter based on the data distribution input by the user includes: Inputting random data into the floating point model to obtain input activation values ​​of the group; Based on the data distribution, statistically analyzing the input activation values ​​of the group to obtain a data range of the input activation values ​​of the group; The target quantization parameter is determined based on the endpoint value of the data range.

8. A post-training quantization device, characterized in that: The device comprises: An acquisition module, configured to acquire, for each group of the floating-point model, model parameters to be adjusted in the group when quantized using a target quantization parameter, wherein the target quantization parameter is a quantization parameter corresponding to the group, and the model parameters to be adjusted include K model parameters in the group that are ranked top in descending order of quantization error; A determination module, used to determine a disturbance tensor corresponding to the to-be-adjusted model parameter when quantized, wherein the disturbance tensor is used to adjust a rounding direction when the to-be-adjusted model parameter is quantized; A quantization module, configured to quantize the model parameters corresponding to the disturbance tensor using the target quantization parameter based on the disturbance tensor deal with.

9. The device according to claim 8, characterized in that The determining module is used to: Acquire a first ratio of the model parameter to be adjusted to a scaling factor included in the target quantization parameter; performing rounding processing on the sum of the first ratio and the disturbance tensor to obtain a first rounding result, and in the case of asymmetric quantization, determining the sum of the first rounding result and the quantization bias included in the target quantization parameter as the quantized value of the model parameter to be adjusted; and in the case of symmetric quantization, determining the first rounding result as the quantized value of the model parameter to be adjusted; Determining a quantization error corresponding to the model parameter to be adjusted based on a quantized value of the model parameter to be adjusted and a value of the model parameter to be adjusted before quantization; Determine a disturbance tensor corresponding to the to-be-adjusted model parameter when the quantization error corresponding to the to-be-adjusted model parameter is minimum.

10. The device according to claim 8 or 9, characterized in that The quantization module is used for: Obtaining a second ratio of a model parameter corresponding to the disturbance tensor to a scaling factor included in the target quantization parameter; The second ratio and the sum of the perturbation tensor are rounded to obtain a second rounded result. In the case of asymmetric quantization, the sum of the second rounded result and the quantization bias included in the target quantization parameter is determined as the quantized value of the model parameter corresponding to the perturbation tensor. In the case of symmetric quantization, the second rounded result is determined as the quantized value of the model parameter corresponding to the perturbation tensor.

11. The device according to any one of claims 8 to 10, characterized in that The model parameters include input activation values ​​or weight values.

12. The device according to any one of claims 8 to 10, characterized in that The model parameters include weight values, and the determination module is further used for: The weight values ​​in the group and the scaling factors corresponding to the weight values ​​in the group are optimized alternately by a single variable method to obtain the scaling factors included in the target quantization parameters and the values ​​of the weight values ​​in the group before quantization.

13. The device according to any one of claims 8 to 10, characterized in that The model parameters include input activation values, the grouping is performed according to the network layer of the floating point model, and the determination module is further used for: The target quantization parameter is determined based on a data set or a data distribution input by a user, wherein the distribution conforming to the data set is the same as the distribution conforming to the training data set of the floating-point model, and the data distribution is the distribution conforming to the input activation values ​​of each network layer of the floating-point model.

14. The device according to claim 13, characterized in that The determining module is further used for: Inputting random data into the floating point model to obtain input activation values ​​of the group; Based on the data distribution, statistically analyzing the input activation values ​​of the group to obtain a data range of the input activation values ​​of the group; The target quantization parameter is determined based on the endpoint value of the data range.

15. A computing device, characterized in that: The computing device comprises a processor and a memory, wherein: The memory stores computer instructions; The processor executes the computer instructions to cause the computing device to perform the method of any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions. When the computer instructions in the computer-readable storage medium are executed by a computing device, the computing device executes the method according to any one of claims 1 to 7.

Images