Data-free post-training quantization method and apparatus, device, and storage medium

Abstract

The present application relates to the technical field of neural networks, and provides a data-free post-training quantization method and apparatus, a device, and a storage medium. The method comprises: acquiring data distribution inputted by a user, wherein the data distribution is distribution to which an input activation value of each network layer of a floating-point model conforms; inputting random data into the floating-point model to obtain the input activation value of each network layer; performing statistical analysis on the input activation value of each network layer on the basis of the data distribution to obtain a data range of the input activation value of each network layer; determining a quantization parameter of the input activation value of each network layer on the basis of endpoint values of the data range; and in the process of using the floating-point model for reasoning, using the quantization parameter of the input activation value of each network layer to perform quantization processing on the input activation value generated in the reasoning process of each network layer. By means of the solution of the present application, input activation values can be quantized without data.

Description

Method, device, equipment and storage medium for quantization after data-free training

[0001] This application claims priority to Chinese patent application No. 2023111850901 filed on September 13, 2023, entitled “Method, device, apparatus and storage medium for quantization after data-free training”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of neural network technology, and in particular to a method, apparatus, device and storage medium for quantization after data-free training. Background Art

[0003] Post-training quantization (PTQ) converts floating-point model parameters to fixed-point parameters after model training. It is a commonly used compression method for deep learning models. During inference, the quantized model can reduce the model's computational complexity, effectively improving inference performance. During post-training quantization, the quantization parameters of the floating-point model's input activation values ​​must be calculated using a standard dataset. However, standard datasets generally involve user privacy and are often unavailable. Therefore, it may not be possible to quantize the floating-point model's input activation values ​​using a standard dataset.

[0004] In the related art, a data-free post-training quantization algorithm based on data generation is provided. In this method, a data generation model is trained by mining information about the training data set in a floating-point model, and the data generation model is used to generate a data set, and the quantization parameters of the input activation value are calculated based on the data set.

[0005] Since a data generation model needs to be trained after training and before quantization, the training overhead is relatively high, so the resource consumption is relatively high.

[0006] Summary of the Invention

[0007] The present application provides a method, apparatus, device and storage medium for quantization after data-free training, which does not require a standard data set or a training data generation model, and can save resources when quantizing a floating-point model.

[0008] In a first aspect, the present application provides a method for quantization after data-free training, the method comprising: obtaining a data distribution of user input, the data distribution being a distribution conforming to the input activation values ​​of each network layer of a floating-point model, inputting random data into the floating-point model to obtain the input activation values ​​of each network layer, performing statistical analysis on the input activation values ​​of each network layer based on the data distribution to obtain a data range of the input activation values ​​of each network layer, determining quantization parameters of the input activation values ​​of each network layer based on the endpoint values ​​of the data range, and performing quantization processing on the input activation values ​​generated during the inference process of each network layer using the quantization parameters of the input activation values ​​of each network layer.

[0009] In the solution shown in this application, the user provides the data distribution of the input activation value, and uses random data as the input of the floating-point model, so as to determine the quantization parameters of the input activation value and quantize the input activation value of the floating-point model. Therefore, there is no need to train the data to generate the model, which can save resource consumption.

[0010] In an optional manner, for each network layer of the floating-point model, the endpoint values ​​of the data range of the input activation value of the network layer are first used to determine the initial value of the quantization parameter of the input activation value of the network layer, and then the initial value of the quantization parameter is used to quantize the input activation value of the network layer to obtain the quantized input activation value, and based on the quantized input activation value and the input activation value of the network layer, the quantization error of the input activation value of the network layer is determined, and the initial value of the quantization parameter of the input activation value of the network layer is updated until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, and the value of the quantization parameter of the input activation value of the network layer is obtained. In this way, using the quantization error as a constraint condition, the value of the quantization parameter can be obtained.

[0011] In an optional manner, when the quantization parameter includes a scaling factor and a quantization bias, the scaling factor and the quantization bias are updated in an alternating update manner, and the processing is as follows: when the scaling factor is an initial value, the quantization bias is updated until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization bias; when the quantization bias update is completed, the scaling factor is updated until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the scaling factor.

[0012] In an optional embodiment, the method can also quantize the weight values ​​in the floating-point model. The process is as follows: for each group of the floating-point model, a quantization parameter corresponding to the group is determined; based on the quantization parameter corresponding to the group, the weight values ​​in the group are quantized to obtain the quantized value of the weight values ​​in the group. This can further reduce the storage resources occupied by the floating-point model and save computing resources. Moreover, determining the quantization parameter by group can quickly determine the quantization parameter.

[0013] In an optional manner, when quantizing the weight values ​​in the floating-point model, the rounding direction of the weight values ​​with large quantization errors is adjusted according to the grouping of the weight values. The process is as follows: determining the perturbation tensor corresponding to the quantization rounding of the weight values ​​to be adjusted in the group, the weight values ​​to be adjusted are the first K weight values ​​in the group whose quantization errors are arranged from large to small during quantization, the perturbation tensor is used to adjust the rounding direction of the weight values ​​to be adjusted during quantization rounding, determining the ratio of the weight values ​​to be adjusted to the corresponding scaling factor; rounding the sum of the ratio and the perturbation tensor to obtain a rounding result, and in the case of asymmetric quantization, updating the sum of the rounding result and the quantization bias corresponding to the weight value to be adjusted to the quantized value of the weight value to be adjusted, and in the case of symmetric quantization, updating the rounding result to the quantized value of the weight value to be adjusted. In this way, the quantization error can be further reduced.

[0014] In an optional manner, the method further includes: grouping the weight values ​​in the floating-point model according to the channels to which the weight values ​​belong, wherein weight values ​​in the same channel belong to one group.

[0015] In a second aspect, the present application provides a device for data-free post-training quantization, which has the function of implementing the above-mentioned first aspect. The device includes at least one module, and the at least one module is used to implement the data-free post-training quantization method provided in the above-mentioned first aspect.

[0016] In some embodiments, the modules in the apparatus for data-free post-training quantization are implemented by software, and the modules in the apparatus are program modules. In other embodiments, the modules in the apparatus are implemented by hardware or firmware.

[0017] In a third aspect, the present application provides a computing device comprising a processor and a memory, wherein: the memory stores computer instructions, and the processor executes the computer instructions to implement the method of the first aspect and its optional implementation methods.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium storing computer instructions. When the computer instructions in the computer-readable storage medium are executed by a computing device, the computing device executes the method of the first aspect and its optional implementation method.

[0019] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the method of the above-mentioned first aspect and its optional implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG1 is a schematic diagram of a system architecture provided by an exemplary embodiment of the present application;

[0021] FIG2 is a schematic diagram of a system architecture provided by an exemplary embodiment of the present application;

[0022] FIG3 is a schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present application;

[0023] FIG4 is a flowchart of a method for post-training quantization provided by an exemplary embodiment of the present application;

[0024] FIG5 is a schematic diagram of a process for quantizing input activation values ​​provided by an exemplary embodiment of the present application;

[0025] FIG6 is a schematic diagram of a flow chart of optimizing a rounding method for weight values ​​provided by an exemplary embodiment of the present application;

[0026] FIG7 is a schematic structural diagram of a device for quantization after data-free training provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0028] The following is an explanation of some terminology concepts involved in the embodiments of this application.

[0029] 1. Model quantization is the process of converting a floating-point model directly into a fixed-point integer model. For example, a floating-point model with a single-precision floating-point (FP32) or half-precision floating-point (FP16) data type is converted into an integer model with an 8-bit integer (int8) data type. Quantized models can reduce the amount of computation required, effectively improving inference speed and reducing memory usage, enabling the deployment of deep learning models on devices with limited memory.

[0030] 2. Post-training quantization is a commonly used deep learning model compression method. After model training, the floating-point parameters in the model are converted to fixed-point parameters. During the conversion, the quantization parameters used need to be calibrated or optimized to ensure that the accuracy loss after model quantization meets the requirements.

[0031] 3. Data-free post-training quantization (PTQ) refers to a method that can directly perform post-training quantization on floating-point models without the user providing a standard dataset.

[0032] The following describes the background of the embodiments of the present application.

[0033] Deep learning has a wide range of applications and practical needs in many fields, such as speech recognition, image processing, and autonomous driving. Some scenarios, such as autonomous driving, have stringent requirements on the latency and memory usage of deep learning model inference. Therefore, deep learning models need to be quantized after training to obtain deep learning models with low latency and low memory usage. During post-training quantization, the quantization parameters of the input activation values ​​in the deep learning model need to be calculated using a standard dataset. However, standard datasets generally involve user privacy and are often unavailable. Therefore, it may not be possible to quantize the input activation values ​​in the deep learning model using a standard dataset. Based on this, the method of data-free post-training quantization has emerged. This method quantizes the model in scenarios where the user does not provide data, and has a wider range of applications than ordinary model quantization methods.

[0034] An embodiment of the present application provides a data-free post-training quantization method, in which a floating-point model can be quantized without data training without the need for a standard data set or a training data generation model, thereby saving resources.

[0035] In the following, the technical solution provided in the embodiment of the present application is described in detail according to the execution subject, system architecture, hardware structure of the device in the system architecture and method flow of the method for quantization after data-free training.

[0036] The following describes the execution of the method for quantization after data-free training.

[0037] The method for performing data-free post-training quantization is performed by a device for performing data-free post-training quantization, referred to as a quantization device. Optionally, the quantization device is a hardware device, such as a computing device such as a terminal device or a server. Alternatively, the quantization device is a software device, such as a software program running on the hardware device.

[0038] The following describes the system architecture provided by the embodiments of the present application.

[0039] In an optional manner, an embodiment of the present application provides a system architecture 100. As shown in Figure 1, the system architecture 100 includes a terminal device 101 and a computing device 102. The terminal device 101 and the computing device 102 are connected via a wired or wireless network. The terminal device 101 is a device used by a user, such as a desktop computer, a laptop computer, or a tablet computer, and the terminal device 101 is used for the user to interact with the computing device 102. For example, the user can input data distribution, etc. through the terminal device 101. The computing device 102 can be a server, etc., and the computing device 102 is used to quantize the pre-trained floating-point model using the data distribution and return the quantized model to the terminal device 101.

[0040] In another optional manner, an embodiment of the present application provides another system architecture 200. As shown in Figure 2, the system architecture 200 includes a terminal device 101 and a public cloud 201. The terminal device 101 is connected to the public cloud 201 via a wired or wireless network. The terminal device 101 is a device used by the user, and the terminal device 101 is used for the user to interact with the public cloud 201. The public cloud 201 is an entity that uses basic resources to provide cloud services to users in a cloud computing model. The public cloud 201 can also be considered as a cloud environment. The public cloud 201 includes a cloud data center. The cloud data center includes a large number of basic resources owned by the cloud service provider. The large number of basic resources include computing resources, storage resources and network resources. The computing resources included in the cloud data center can be a computing device cluster. The computing device cluster includes at least one computing device 102. The computing device 102 can be a server, etc. When a user uses a cloud service, the user can input data distribution to the public cloud 201 through an API or an interactive interface (the interactive interface can be a graphical user interface (GUI)). The computing device in the public cloud 201 receives the pre-trained floating-point model and data distribution input by the user, uses the data distribution to quantize the pre-trained floating-point model, and returns the quantized model to the terminal device 101.

[0041] In the system architecture 200, the method of post-quantization without data training can be abstracted by the cloud service provider into a cloud service in the public cloud 201 and provided to users. After the user obtains the permission to use the cloud service, he can use the cloud service to perform post-quantization without data training.

[0042] In yet another optional embodiment, the present application further provides a system architecture. The system architecture includes a terminal device, which is a device used by a user. The terminal device executes the method for post-dataless training quantization in the present application embodiment.

[0043] The following is an introduction to the hardware structure of the device.

[0044] As shown in Figure 3, device 300 is optionally implemented by a general bus architecture and includes at least one processor 301, a communication bus 302, a memory 303, and at least one network interface 304. Device 300 is the terminal device 101 and computing device 102 mentioned above.

[0045] The processor 301 is, for example, a general-purpose central processing unit (CPU), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the solution of the present application. For example, the processor 301 includes an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0046] Communication bus 302 is used to transmit information between the above components. Communication bus 302 can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 3, but this does not mean that there is only one bus or one type of bus.

[0047] The memory 303 is, for example, a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 303 is, for example, independent and connected to the processor 301 via the communication bus 302. The memory 303 can also be integrated with the processor 301.

[0048] Optionally, the memory 303 is used to store a pre-trained floating-point model. When the device 300 needs to use the floating-point model, the processor 301 accesses the memory 303 to obtain the floating-point model stored in the memory 303.

[0049] The network interface 304 uses any transceiver-like device for communicating with other devices or communication networks. The network interface 304 includes a wired network interface and may also include a wireless network interface. For example, the wired network interface may be an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless network interface may be a wireless local area network (WLAN) interface, a cellular network interface, or a combination thereof.

[0050] In a specific implementation, as an example, the processor 301 may include one or more CPUs.

[0051] In a specific implementation, as an example, the device 300 may include multiple processors. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0052] In some embodiments, the memory 303 is used to store program code 3031 for executing the data-free training and post-quantization method in the present application, and the processor 301 executes the program code 3031 stored in the memory 303. That is, the device 300 can implement the data-free training and post-quantization method provided in the method embodiment through the processor 301 and the program code 3031 in the memory 303.

[0053] The following describes the post-training quantization method flow with reference to FIG4 to FIG6 .

[0054] In post-training quantization, the quantized value of the weight value can be expressed as W q =W+ΔW, where W represents the value of the weight value before quantization and ΔW represents the difference between the weight value before and after quantization. The value of the input activation value after quantization is represented as A q =A+ΔA, where A represents the value of the input activation value before quantization, ΔA represents the difference between the input activation value before and after quantization, and the final calculated value after quantization is expressed as formula (1). q A q =WA+AΔW+WΔA+ΔWΔA (1)

[0055] The goal of quantization is to minimize the quantization error, which is expressed as formula (2), or the goal of quantization is to make the quantization error less than or equal to a certain value.

[0056] In the embodiment of the present application, quantization error is used as the optimization target. Combining formula (1) and formula (2), it can be seen that the quantization error mainly comes from the quantization of input activation values ​​and the quantization of weight values. Therefore, in the embodiment of the present application, in response to the quantization requirements of the no-data scenario, input activation value quantization and weight value quantization are performed separately to minimize the quantization error, or to achieve a value less than or equal to a certain value.

[0057] As can be seen from the above description, post-training quantization includes two parts: input activation value quantization and weight value quantization. Referring to Figure 4, when the input activation value is quantized, the input of post-training quantization includes a pre-trained floating-point model and data distribution. The pre-trained floating-point model is a deep learning model that has been trained. The data type of the parameters in the deep learning model is floating-point. The deep learning model includes deep learning models for scenarios such as speech recognition, text recognition, image processing, or vehicle driving. The embodiments of the present application are not limited. The data type in the pre-trained floating-point model is any floating-point type, for example, FP32, FP16, or FP64. The floating-point model includes multiple network layers connected in sequence. For example, the network layer of the floating-point model may include a convolutional layer, a fully connected layer, and a pooling layer. The network layers included in different floating-point models may be different. The multiple network layers included in the floating-point model correspond to the application scenarios of the floating-point model. For example, the floating-point model is a deep learning model for text recognition, and the multiple network layers include an embedding layer, a convolutional layer, and a fully connected layer. The data distribution is the distribution statistics of the input activation value. For example, the data distribution is a normal distribution. When quantizing weight values, the input to the training quantization includes a pre-trained floating-point model. There is no order in which the input activation value quantization and the weight value quantization are performed. The input activation value can be quantized first, or the weight value can be quantized first. This is not limited in the embodiments of the present application. FIG4 shows that the input activation value is quantized first, and then the weight value is quantized. After the input activation value quantization and the weight value quantization, the quantized model is output.

[0058] The following describes the process of input activation value quantization and the process of weight value quantization respectively.

[0059] 1. Activation value quantization process. This process includes the quantization of the input activation value and the optimization of the rounding method.

[0060] 1.1. The quantization process of the input activation value is shown in steps 501 to 505 in FIG. 5 . The process shown in FIG. 5 is described using the system architecture shown in FIG. 2 as an example. The computing device in FIG. 5 is the computing device 102 described above.

[0061] Step 501: Obtain data distribution input by the user.

[0062] In this embodiment, the user inputs data distribution through the terminal device, and the computing device receives the data distribution sent by the terminal device. The data distribution is the distribution conforming to the input activation values ​​of each network layer of the floating-point model, that is, the input activation values ​​of each network layer conform to the data distribution, such as the data distribution is a normal distribution or a Poisson distribution.

[0063] In one optional approach, if the pre-trained floating-point model is obtained by training the computing device, the user may not provide the floating-point model. This is equivalent to the computing device training the floating-point model and then providing the user with an input interface. The user uploads the data distribution of input activation values ​​through the input interface, and the computing device receives the data distribution. If the pre-trained floating-point model is not obtained by training the computing device, the computing device provides the user with an input interface. The user uploads the pre-trained floating-point model and the data distribution of input activation values ​​through the input interface.

[0064] In an optional manner, the user further inputs a quantization bit number n through the terminal device, where the quantization bit number represents the number of bits of each input activation value after quantization, and the computing device obtains the quantization bit number n.

[0065] Step 502: random data is input into the floating-point model to obtain input activation values ​​of each network layer.

[0066] In this embodiment, random data is stored in the computing device. The random data is commonly used data that can make the input activation value of each network layer conform to the data distribution of the user input. Alternatively, random data is stored in the computing device. The random data corresponds to a data distribution. The computing device obtains random data that conforms to the data distribution of the user input from the stored random data. Alternatively, the computing device generates random data that conforms to the data distribution based on the data distribution. The computing device inputs the random data into the floating-point model and performs forward reasoning to obtain the input activation value of each network layer of the floating-point model. For the network layer that is the frontmost among the multiple network layers connected in sequence in the floating-point model, the input activation value of the network layer can be a feature of the random data. For any network layer other than the frontmost network layer among the multiple network layers, the input activation value of the network layer is the feature data output by the previous network layer located before the network layer.

[0067] Step 503: Based on the data distribution, statistical analysis is performed on the input activation values ​​of each network layer to obtain the data range of the input activation values ​​of each network layer.

[0068] In this embodiment, the data distribution of the user input is the data distribution that the input activation value of each network layer conforms to. For any network layer, the computing device uses this data distribution to determine the maximum and minimum values ​​of the input activation value of the network layer, and obtains the data range of the input activation value. For example, if the data distribution is a normal distribution, for the kth network layer of the floating-point model, the input activation value of the kth network layer can be used to calculate the parameters μ and σ of the normal distribution, where μ is the mathematical expectation and σ is the standard deviation. In the normal distribution, the probability that the data falls within (μ-3σ, μ+3σ) is 99.74%, so (μ-3σ, μ+3σ) can be used as the data range of the input activation value of the kth network layer, where μ-3σ is the minimum value of the data range and μ+3σ is the maximum value of the data range.

[0069] Combined with the given quantization bit number n, the range of the quantized input activation value is [-2 n-1 , 2 n-1 -1].

[0070] Step 504: Determine the quantization parameters of the input activation values ​​of each network layer based on the endpoint values ​​of the data range.

[0071] In this embodiment, for any network layer, the endpoint values ​​of the data range of the network layer are the maximum and minimum values ​​of the input activation value of the network layer, and the computing device uses the maximum and minimum values ​​to determine the quantization parameters of the input activation value of the network layer. In the case of asymmetric quantization, the quantization parameters include a scaling factor and a quantization bias, and in the case of symmetric quantization, the quantization parameters include a scaling factor.

[0072] In an optional manner, the process of determining the quantization parameter using the endpoint values ​​of the data range is as follows:

[0073] In the case of asymmetric quantization, the quantization parameters include a scaling factor and a quantization bias. For any network layer, the computing device uses formula (3) to calculate the initial value of the scaling factor corresponding to the network layer, and uses formula (4) to calculate the initial value of the quantization bias corresponding to the network layer.

[0074] In formula (3) and formula (4), scale1 represents a scaling factor, max_val represents the maximum value of the input activation value of the network layer, min_val represents the minimum value of the input activation value of the network layer, n represents the number of quantization bits, offset1 represents the quantization bias, and round represents a rounding operation, which can be a rounding operation, an upward rounding operation, or a downward rounding operation.

[0075] Then, the computing device uses the initial value of the quantization parameter and uses formula (5) to quantize the input activation value of the network layer to obtain the quantized value of the input activation value, dequantizes the quantized value of the input activation value to obtain the dequantized value, and uses formula (6) or (7) to calculate the quantization error of the input activation value of the network layer. It is judged whether the quantization error is less than or equal to the target value corresponding to the network layer. If the quantization error is less than or equal to the target value, the initial value of the scaling factor is determined as the value of the scaling factor, and the initial value of the quantization bias is determined as the value of the quantization bias. If the quantization error is greater than the target value, the initial value of the quantization parameter of the input activation value of the network layer is updated, and the above process is continued until the quantization error of the input activation value of the network layer is less than or equal to the target value, and the value of the quantization parameter of the input activation value of the network layer is obtained. Alternatively, the initial value of the quantization parameter of the input activation value of the network layer is updated until the quantization error of the input activation value of the network layer is minimized, and the value of the quantization parameter of the input activation value of the network layer is obtained. Among them, the target value can be set based on the actual situation. The larger the number of input activation values ​​of the network layer, the larger the target value is set, and the smaller the number of input activation values ​​of the network layer, the smaller the target value is set.

[0076] In formula (5), A2 represents the value of the input activation value after quantization, A1 represents the value of the input activation value before quantization, scale1 represents the scaling factor, offset1 represents the quantization bias, and round represents the rounding operation.

[0077] In formula (6), L(A1, A2) represents the quantization error, A 1i Represents the value of the i-th input activation value before quantization, i is an integer less than or equal to m, A 2i It represents the dequantized value of the quantized value of the i-th input activation value. The dequantized value is a floating point number. m represents the number of input activation values ​​of the network layer.

[0078] Optionally, in the case of asymmetric quantization, the computing device may use an alternating update method to determine the scaling factor and the quantization bias. For example, for any network layer, when the scaling factor is an initial value, the input activation value of the network layer is quantized using the initial value of the scaling factor and the initial value of the quantization bias to obtain the quantized value of the input activation value of the network layer, see formula (5).

[0079] The computing device then dequantizes the quantized value and uses formula (6) or (7) to calculate the quantization error corresponding to the network layer. The computing device increases the initial value of the quantization bias by the first adjustment amplitude, substitutes the initial value of the scaling factor and the adjusted quantization bias into formula (5), quantizes the input activation value of the network layer, obtains the quantized value of the input activation value of the network layer, and uses formula (6) or (7) to calculate the quantization error corresponding to the network layer. The computing device determines whether the quantization error meets the minimum or is less than or equal to the target value. If not, the quantization bias is continuously adjusted, and so on, until the value of the quantization bias is obtained when the quantization error is minimum, or until the value of the quantization bias is obtained when the quantization error is less than or equal to the target value.

[0080] Then, after obtaining the quantization bias, fix the quantization bias and update the scaling factor to determine the value of the scaling factor when the quantization error is minimized, or the value of the scaling factor when the quantization error is less than or equal to the target value. The process of updating the scaling factor is similar to the process of updating the quantization bias and will not be repeated here.

[0081] In this way, after multiple alternating updates of the quantization bias and the scaling factor, the final scaling factor and quantization bias are obtained.

[0082] In the case of symmetric quantization, the target quantization parameter includes a scaling factor. For any network layer, the computing device uses formula (3) to calculate the initial value of the scaling factor corresponding to the network layer. For any network layer, only the scaling factor corresponding to the network layer is adjusted, which is equivalent to setting the quantization bias to 0. The process of determining the scaling factor refers to the process of asymmetric quantization and is not repeated here.

[0083] Step 505 , during the inference process of the floating-point model, the quantization parameters of the input activation values ​​of each network layer are used to quantize the input activation values ​​generated during the inference process of each network layer.

[0084] In this embodiment, for each network layer, the quantization parameters of all input activation values ​​of the network layer are the same. In the subsequent inference process using the floating-point model, after the inference data is input into the floating-point model, the input activation values ​​of each network layer are obtained, and the quantization parameters of the input activation values ​​of the network layer are used to quantize the input activation values ​​of the network layer. Then, the inference calculation is performed. In this way, the floating-point input activation values ​​are converted into integer input activation values, which can reduce the amount of calculation.

[0085] The device that performs step 505 can be the same device as or different from the device that performs steps 501 to 504. For example, the computing device determines the quantization parameter of the input activation value and performs the inference process. For another example, after obtaining the quantization parameter of the input activation value based on the process shown in FIG5 , the computing device returns the quantization parameter to the terminal device, which stores the quantization parameter. Subsequently, during the inference process, the terminal device uses the quantization parameter to quantize the input activation value, thereby reducing the amount of calculation of the input activation value.

[0086] Using the process shown in Figure 5, when performing model quantization, users do not need to provide data, which protects user privacy and has a wider range of applications. It also does not require additional training to obtain the data generator, saving resources and improving quantization efficiency.

[0087] In data-free post-training quantization scenarios, initializing the quantization parameters of the input activation values ​​based on their data distribution yields higher accuracy than random estimation. Furthermore, in data-free post-training quantization scenarios, there are no requirements for the floating-point model structure, leading to stronger generalization capabilities.

[0088] 1.2. Optimization process of rounding method during input activation value quantization.

[0089] After determining the quantization parameters, for the input activation value of any network layer, when no perturbation tensor is introduced, the input activation value of the network layer is quantized and rounded to obtain the quantized value of the input activation value (quantization is performed using formula (5). In the case of symmetric quantization, the quantization bias is 0). The quantization error of the input activation value of the network layer is calculated using formula (6) or (7).

[0090] Then determine the K input activation values ​​in the network layer that are ranked from largest to smallest in terms of quantization error, introduce a perturbation tensor for these K input activation values ​​(see formula (8) for asymmetric quantization and formula (9) for symmetric quantization), quantize and round the input activation values ​​of the network layer to obtain the quantized values ​​of the input activation values, continue to use formula (6) or (7) to calculate the quantization error of the input activation values ​​to be adjusted of the network layer (m represents the number of input activation values ​​to be adjusted), update the perturbation tensor until the quantization error is minimized, and obtain the perturbation tensor with the minimum quantization error. Alternatively, calculate whether the quantization error of each input activation value to be adjusted is the minimum. If it is the minimum, obtain the perturbation tensor corresponding to the input activation value to be adjusted. If it is not the minimum, update the perturbation tensor until the quantization error of each input activation value to be adjusted is the minimum, and obtain the perturbation tensor with the minimum quantization error. The perturbation tensors corresponding to the K input activation values ​​of the network layer can be the same, partially the same, or completely different.

[0091] Wherein, in formulas (8) and (9), Δ1 is the perturbation tensor.

[0092] In addition, the principle of introducing the perturbation tensor is: assuming that the rounding operation is rounding, the decimal part of the original ratio of the input activation value to the scaling factor is less than 5. By introducing the perturbation tensor, the decimal part is changed to be greater than or equal to 5, thereby changing the rounded value, or, the decimal part of the original ratio of the input activation value to the scaling factor is greater than or equal to 5. By introducing the perturbation tensor, the decimal part is changed to be less than 5, thereby changing the rounded value.

[0093] It should be noted that after determining the quantization parameters, the perturbation tensor is introduced for the input activation values ​​with relatively large quantization errors in each network layer, and the rounding direction is further adjusted to further reduce the quantization error.

[0094] 2. The process of weight value quantization: The process of weight value quantization includes the process of determining the quantization process of the weight value and the process of optimizing the rounding method.

[0095] 2.1. The quantization process of weight value is:

[0096] The weight values ​​in the floating-point model are divided into a plurality of groups, each group including one or more weight values. For the weight values ​​in each group of the floating-point model, the quantization parameters corresponding to the weight values ​​in the group are the same.

[0097] In an optional manner, in the case of asymmetric quantization, the quantization parameter corresponding to the weight value in the group is determined according to formula (10), and in the case of symmetric quantization, the quantization parameter corresponding to the weight value in the group is determined according to formula (11).

[0098] Where L(W1, W2) represents the quantization error of the weight value in a group. The expression is shown in formula (12). In formula (10), When minimizing the quantization error, the values ​​of the scaling factor scale2 and the quantization bias offset2 are determined using formula (13). In formula (11), When minimizing the quantization error, the value of scale2 and W2 are determined using formula (14).

[0099] In formula (12) and formula (15), W1 represents the value of the weight value before quantization, W2 represents the value of the weight value after quantization and the value after dequantization, W 1j Represents the value of the jth weight value before quantization, j is an integer less than or equal to g, W2j represents the dequantized value of the jth weight value after quantization, and g represents the number of weight values ​​in the group. In formulas (13) and (14), scale2 represents the scaling factor of the weight value, offset2 represents the quantization bias of the weight value, round represents the rounding operation, which can be rounding, rounding up, or rounding down. Δ2 represents the perturbation tensor described later, which is 0 here.

[0100] In formula (10), two parameters, the scaling factor scale2 and the quantization offset offset2, need to be adjusted. When adjusting these two parameters, the initial values ​​of the two parameters are preset values. Under the premise of fixing scale2, adjust offset2 so that the quantization error L(W1, W2) of the weight value in the group is minimized. Then, fix offset2 to the offset2 when L(W1, W2) is minimized, and adjust scale2 so that L(W1, W2) is minimized. The process is executed a preset number of times, and the scale2 and offset2 at the last execution of the process are determined as the final quantization parameters. The preset number is set according to an empirical value, such as 10.

[0101] In another optional manner, when adjusting the quantization parameter of the weight value, when L(W1, W2) is less than or equal to the target value, it can also be considered that the quantization parameter is determined to be obtained.

[0102] For the weight value in each group, after the quantization parameter is determined, quantization processing is performed using the quantization parameter to obtain a quantized value of the weight value.

[0103] 2.2. Rounding method optimization and quantification process based on the optimized rounding method.

[0104] After determining the quantization parameters of the weight values ​​in each group, the quantization errors of some weight values ​​in each group may be relatively large. In order to further reduce the quantization errors of these weight values, a quantization process based on the optimized rounding method is performed, referring to steps 601 to 603 in FIG6 . The process shown in FIG6 is explained using the system architecture shown in FIG2 in the previous text as an example, and the computing device in FIG6 is the computing device 102 in the previous text.

[0105] Step 601: For each weight value in the floating-point model, determine the disturbance tensor corresponding to the quantization and rounding of the weight value to be adjusted in the group.

[0106] In this embodiment, for the weight values ​​of any group, when no perturbation tensor is introduced, in the case of asymmetric quantization, the quantized value of the weight value is calculated using formula (13) (Δ2 is 0), and in the case of symmetric quantization, the quantized value of the weight value is calculated using formula (14) (Δ2 is 0). The weight values ​​in the group are quantized and rounded to obtain the quantized value of the weight value, and the quantization error of the weight value in the group is calculated using formula (12) or (15).

[0107] The quantization errors of the weight values ​​in the group are then sorted from largest to smallest, and the weight values ​​of the top K quantization errors are determined. The weight values ​​of these K quantization errors are the weight values ​​to be adjusted. The value of K may vary for different groups. The more weight values ​​in the group, the larger the value of K. Conversely, the fewer weight values ​​in the group, the smaller the value of K.

[0108] Then, a perturbation tensor is introduced for the weight values ​​to be adjusted in the group. In the case of asymmetric quantization, the quantized value of the weight values ​​to be adjusted is calculated using formula (13). In the case of symmetric quantization, the quantized value of the weight values ​​to be adjusted is calculated using formula (14). Continue to use formula (12) or (15) to calculate the quantization error of the weight values ​​to be adjusted (g represents the number of weight values ​​to be adjusted) until a suitable perturbation tensor is selected to minimize the quantization error. This process is represented by formula (16). Alternatively, calculate whether the quantization error of each weight value to be adjusted is the smallest. If it is the smallest, obtain the perturbation tensor corresponding to the weight value to be adjusted. If it is not the smallest, update the perturbation tensor until the quantization error of each weight value to be adjusted is minimized, and obtain the perturbation tensor with the smallest quantization error. For any group, the perturbation tensors corresponding to the weight values ​​to be adjusted in the group can be the same, partially the same, or completely different.

[0109] In formula (16), The value of Δ2 when minimizing the quantization error, and the expression of L(W1, W2) can be found in formula (12) or (15).

[0110] It should be noted that the principle of introducing the perturbation tensor is: assuming that the rounding operation is rounding, the decimal part of the original ratio of the weight value to the scaling factor is less than 5. By introducing the perturbation tensor, the decimal part is changed to be greater than or equal to 5, thereby changing the rounded value; or, the decimal part of the original ratio of the weight value to the scaling factor is greater than or equal to 5. By introducing the perturbation tensor, the decimal part is changed to be less than 5, thereby changing the rounded value.

[0111] Step 602: Determine the ratio of the weight value to be adjusted to the corresponding scaling factor.

[0112] Step 603, rounding the ratio and the sum of the perturbation tensor to obtain a rounding result. In the case of asymmetric quantization, the sum of the rounding result and the quantization bias corresponding to the weight value to be adjusted is updated to the quantized value of the weight value to be adjusted. In the case of symmetric quantization, the rounding result is updated to the quantized value of the weight value to be adjusted.

[0113] In this embodiment, in the case of asymmetric quantization, the quantized value of the weight value to be adjusted is calculated using formula (13), and in the case of symmetric quantization, the quantized value of the weight value to be adjusted is calculated using formula (14).

[0114] In formulas (13) and (14), Indicates the ratio of the weight value to be adjusted to the scaling factor.

[0115] The weight values ​​to be adjusted in any group can be quantized using the process shown in FIG6 .

[0116] In an optional manner, there are multiple ways to group weight values ​​in a floating-point model, and two feasible ways are provided below.

[0117] Method 1: Group the weight values ​​in the floating-point model according to the channels to which the weight values ​​belong. The weight values ​​in the same channel are grouped into the same group.

[0118] For example, the weight value of the convolution layer of the floating-point model is a four-dimensional tensor, which is expressed as s×e×h×r. s represents the number of output channels of the convolution layer, e represents the number of input channels of the convolution layer, and h×r represents the h×r convolution kernel. During quantization, e×h×r weight values ​​are a group.

[0119] Method 2: The weight values ​​in the floating-point model are grouped according to the network layer to which the weight values ​​belong. The weight values ​​belonging to the same network layer are grouped into the same group.

[0120] In an embodiment of the present application, when quantizing the weight value, the quantization error is minimized or less than or equal to the target value as a constraint condition. Quantizing the weight value can reduce the error caused by quantization and improve the inference accuracy of the quantized model.

[0121] In the embodiments of this application, products that may be involved include model compression tools or inference model deployment tools. Subsequent infringement determinations can be made using the following methods:

[0122] 1. Check whether input data is required from the quantized input after training and whether the data distribution of input activation values ​​is required.

[0123] 2. Analyze the model compression process and logs to determine whether the quantization process uses a dataset, whether additional data processing steps are performed, and whether it is necessary to initialize different random data for different data distributions to calculate the quantization parameters of the input activation values.

[0124] 3. Check the functions released by the model compression tool to see whether it integrates the above-mentioned process of quantization after data-free training.

[0125] FIG7 is a block diagram of a device for quantization after data-free training provided in an embodiment of the present application. The device can be implemented as part or all of the device through software, hardware, or a combination of both. The device provided in an embodiment of the present application can implement the process described in FIG4 to FIG6 of the embodiment of the present application, and the device includes: an acquisition module 710 and a quantization module 720, wherein:

[0126] Acquisition module 710 is used to obtain the data distribution of the user input, where the data distribution is the distribution of the input activation values ​​of each network layer of the floating-point model, and can be used to implement the acquisition function of step 501 and execute the implicit steps included in step 501;

[0127] The quantization module 720 is configured to:

[0128] Inputting random data into the floating-point model to obtain input activation values ​​of each network layer;

[0129] Based on the data distribution, statistically analyzing the input activation values ​​of each network layer to obtain a data range of the input activation values ​​of each network layer;

[0130] Determining quantization parameters of input activation values ​​of each network layer based on endpoint values ​​of the data range;

[0131] During the inference process using the floating-point model, the quantization parameters of the input activation values ​​of each network layer are used to quantize the input activation values ​​generated during the inference process of each network layer, which can be specifically used to implement the input activation value quantization function of steps 502 to 505 and execute the implicit steps contained in steps 502 to 505.

[0132] In an optional manner, the quantization module 720 is configured to:

[0133] For each network layer of the floating-point model, determining an initial value of a quantization parameter of the input activation value of the network layer using an endpoint value of a data range of the input activation value of the network layer;

[0134] quantizing the input activation value of the network layer using the initial value of the quantization parameter to obtain a quantized input activation value;

[0135] Determining a quantization error of the input activation value of the network layer based on the quantized input activation value and the input activation value of the network layer;

[0136] Update the initial value of the quantization parameter of the input activation value of the network layer until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, to obtain the value of the quantization parameter of the input activation value of the network layer.

[0137] In an optional manner, the quantization parameter includes a scaling factor and a quantization bias;

[0138] The quantization module 720 is configured to:

[0139] When the scaling factor is an initial value, updating the quantization bias until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, to obtain the value of the quantization bias;

[0140] When the quantization bias update is completed, the scaling factor is updated until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, to obtain the value of the scaling factor.

[0141] In an optional manner, the quantization module 720 is further configured to:

[0142] For each group of the floating-point model, determine a quantization parameter corresponding to the group, wherein in the case of asymmetric quantization, the quantization parameter corresponding to the group includes a scaling factor and a quantization bias, and in the case of symmetric quantization, the quantization parameter corresponding to the group includes a scaling factor;

[0143] Based on the quantization parameter corresponding to the group, the weight value in the group is quantized to obtain a quantized value of the weight value in the group.

[0144] In an optional manner, the quantization module 720 is further configured to:

[0145] Determine a disturbance tensor corresponding to quantization rounding of the weight values ​​to be adjusted in the group, wherein the weight values ​​to be adjusted are the first K weight values ​​in the group arranged from largest to smallest in terms of quantization error during quantization, and the disturbance tensor is used to adjust the rounding direction of the weight values ​​to be adjusted during quantization rounding;

[0146] Determining a ratio of the weight value to be adjusted to a corresponding scaling factor;

[0147] The ratio and the sum of the perturbation tensor are rounded to obtain a rounded result. In the case of asymmetric quantization, the sum of the rounded result and the quantization bias corresponding to the weight value to be adjusted is updated to the quantized value of the weight value to be adjusted. In the case of symmetric quantization, the rounded result is updated to the quantized value of the weight value to be adjusted.

[0148] In an optional manner, the acquisition module 710 is configured to:

[0149] The weight values ​​in the floating-point model are grouped according to the channels to which the weight values ​​belong, wherein weight values ​​in the same channel belong to one group.

[0150] The detailed process of quantizing input activation values ​​by the apparatus for quantizing after data-free training shown in FIG7 is described in the previous embodiments and will not be repeated here. The apparatus for quantizing after data-free training shown in FIG7 is the computing device 102 in FIG1 and FIG2.

[0151] In an optional embodiment, a computer program product is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computing device to execute the processes shown in FIG4 to FIG6.

[0152] Those skilled in the art will appreciate that the various method steps and units described in the embodiments disclosed in this application can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed system architecture, device and method can be implemented in other ways. For example, the device embodiment described above is only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.

[0154] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0155] In addition, each module in each embodiment of the present application can be integrated into a processing module, each module can exist physically separately, or two or more modules can be integrated into a module. The above-mentioned integrated modules can be implemented in the form of hardware or software modules.

[0156] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0157] The term "plurality" in this application means two or more.

[0158] The above description is merely an exemplary embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for quantization after data-free training, characterized in that: The method comprises: Obtaining data distribution of user input, where the data distribution is a distribution conforming to input activation values ​​of each network layer of the floating-point model; Inputting random data into the floating point model to obtain input activation values ​​of each network layer; Based on the data distribution, statistically analyzing the input activation values ​​of each network layer to obtain the data range of the input activation values ​​of each network layer; Determining quantization parameters of input activation values ​​of each network layer based on endpoint values ​​of the data range; During the inference process using the floating-point model, the input activation values ​​generated during the inference process of each network layer are quantized using the quantization parameters of the input activation values ​​of each network layer.

2. The method according to claim 1, characterized in that The step of determining the quantization parameters of the input activation values ​​of each network layer based on the endpoint values ​​of the data range includes: For each network layer of the floating-point model, using the endpoint values ​​of the data range of the input activation values ​​of the network layer, determining the initial value of the quantization parameter of the input activation value of the network layer; Using the initial value of the quantization parameter, quantizing the input activation value of the network layer to obtain a quantized input activation value; Determining a quantization error of the input activation value of the network layer based on the quantized input activation value and the input activation value of the network layer; Update the initial value of the quantization parameter of the input activation value of the network layer until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization parameter of the input activation value of the network layer.

3. The method according to claim 2, characterized in that The quantization parameters include a scaling factor and a quantization bias; The updating of the initial value of the quantization parameter of the input activation value of the network layer until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization parameter of the input activation value of the network layer, includes: When the scaling factor is an initial value, updating the quantization bias until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization bias; When the quantization bias update is completed, the scaling factor is updated until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the scaling factor.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: For each group of the floating-point model, determine a quantization parameter corresponding to the group, wherein in the case of asymmetric quantization, the quantization parameter corresponding to the group includes a scaling factor and a quantization bias, and in the case of symmetric quantization, the quantization parameter corresponding to the group includes a scaling factor; Based on the quantization parameter corresponding to the group, the weight value in the group is quantized to obtain the quantized value of the weight value in the group.

5. The method according to claim 4, characterized in that The method further comprises: Determine a disturbance tensor corresponding to quantization rounding of the weight value to be adjusted in the group, wherein the weight value to be adjusted is the first K weight values ​​arranged from large to small in the group when quantization errors are quantized, and the disturbance tensor is used to adjust the rounding direction of the weight value to be adjusted when quantization rounding; Determining a ratio of the weight value to be adjusted to a corresponding scaling factor; The ratio and the sum of the disturbance tensor are rounded to obtain a rounded result. In the case of asymmetric quantization, the sum of the rounded result and the quantization bias corresponding to the weight value to be adjusted is updated to the quantized value of the weight value to be adjusted. In the case of symmetric quantization, the rounded result is updated to the quantized value of the weight value to be adjusted.

6. The method according to claim 4 or 5, characterized in that: The method further comprises: The weight values ​​in the floating-point model are grouped according to the channels to which the weight values ​​belong, wherein the weight values ​​in the same channel belong to one group.

7. A device for quantization after data-free training, characterized in that: The device comprises: An acquisition module, used to acquire data distribution of user input, where the data distribution is the distribution conforming to the input activation values ​​of each network layer of the floating-point model; Quantization module for: Inputting random data into the floating point model to obtain input activation values ​​of each network layer; Based on the data distribution, statistically analyzing the input activation values ​​of each network layer to obtain the data range of the input activation values ​​of each network layer; Determining quantization parameters of input activation values ​​of each network layer based on endpoint values ​​of the data range; During the inference process using the floating-point model, the input activation values ​​generated during the inference process of each network layer are quantized using the quantization parameters of the input activation values ​​of each network layer.

8. The device according to claim 7, characterized in that The quantization module is used for: For each network layer of the floating-point model, using the endpoint values ​​of the data range of the input activation values ​​of the network layer, determining the initial value of the quantization parameter of the input activation value of the network layer; Using the initial value of the quantization parameter, quantizing the input activation value of the network layer to obtain a quantized input activation value; Determining a quantization error of the input activation value of the network layer based on the quantized input activation value and the input activation value of the network layer; Update the initial value of the quantization parameter of the input activation value of the network layer until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization parameter of the input activation value of the network layer.

9. The device according to claim 8, characterized in that The quantization parameters include a scaling factor and a quantization bias; The quantization module is used for: When the scaling factor is an initial value, updating the quantization bias until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization bias; When the quantization bias update is completed, the scaling factor is updated until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the scaling factor.

10. The device according to any one of claims 7 to 9, characterized in that The quantization module is further used for: For each group of the floating-point model, determine a quantization parameter corresponding to the group, wherein in the case of asymmetric quantization, the quantization parameter corresponding to the group includes a scaling factor and a quantization bias, and in the case of symmetric quantization, the quantization parameter corresponding to the group includes a scaling factor; Based on the quantization parameter corresponding to the group, the weight value in the group is quantized to obtain the quantized value of the weight value in the group.

11. The device according to claim 10, characterized in that The quantization module is further used for: Determine a disturbance tensor corresponding to quantization rounding of the weight value to be adjusted in the group, wherein the weight value to be adjusted is the first K weight values ​​arranged from large to small in the group when quantization errors are quantized, and the disturbance tensor is used to adjust the rounding direction of the weight value to be adjusted when quantization rounding; Determining a ratio of the weight value to be adjusted to a corresponding scaling factor; The ratio and the sum of the disturbance tensor are rounded to obtain a rounded result. In the case of asymmetric quantization, the sum of the rounded result and the quantization bias corresponding to the weight value to be adjusted is updated to the quantized value of the weight value to be adjusted. In the case of symmetric quantization, the rounded result is updated to the quantized value of the weight value to be adjusted.

12. The device according to claim 10 or 11, characterized in that The acquisition module is used to: The weight values ​​in the floating-point model are grouped according to the channels to which the weight values ​​belong, wherein the weight values ​​in the same channel belong to one group.

13. A computing device, characterized in that: The computing device comprises a processor and a memory, wherein: The memory stores computer instructions; The processor executes the computer instructions to cause the computing device to perform the method of any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions. When the computer instructions in the computer-readable storage medium are executed by a computing device, the computing device executes the method according to any one of claims 1 to 6.

Images