Quantification method and device after data-free training, equipment and storage medium

Abstract

The invention provides a method, a device and equipment for quantization after data-free training, and a storage medium, and belongs to the technical field of neural networks. The method comprises the steps of obtaining data distribution input by a user, inputting random data into the floating point model so as to obtain the input activation value of each network layer, performing statistical analysis on the input activation value of each network layer based on the data distribution, and obtaining the input activation value of each network layer according to the statistical analysis result. The method comprises the following steps: acquiring a data range of an input activation value of each network layer, determining a quantization parameter of the input activation value of each network layer based on an endpoint value of the data range, and performing quantization processing on the input activation value generated in the reasoning process of each network layer by using the quantization parameter of the input activation value of each network layer in the reasoning process of using a floating point model. By adopting the scheme of the invention, the input activation value can be quantized under the condition of no data.

Description

Technical Field

[0001] The present disclosure relates to the field of neural network technology, and in particular to a method, device, equipment and storage medium for quantization after data-free training. Background Art

[0002] Post-training quantization (PTQ) is a commonly used deep learning model compression method that converts the floating-point parameters in the model into fixed-point parameters after model training. During inference, the quantized model can reduce the amount of model computation and effectively improve inference performance. When performing post-training quantization, the quantization parameters of the input activation values ​​in the floating-point model need to be calculated using a standard data set. However, standard data sets generally involve user privacy and are often not available. Therefore, it may not be possible to use standard data sets to quantize the input activation values ​​in the floating-point model.

[0003] In the related art, a data-free post-training quantization algorithm based on data generation is provided. In this method, a data generation model is trained by mining information about the training data set in a floating-point model, and the data set is generated using the data generation model. The quantization parameters of the input activation values ​​are calculated based on the data set.

[0004] Since a data generation model needs to be trained after training and before quantization, the training overhead is relatively large, so the resource consumption is relatively high. Summary of the invention

[0005] The present disclosure provides a method, apparatus, device and storage medium for quantization after data-free training, which does not require a standard data set or a training data generation model, and can save resources when quantizing a floating-point model.

[0006] In a first aspect, the present disclosure provides a method for quantization after data-free training, the method comprising: obtaining a data distribution of user input, the data distribution being a distribution conforming to the input activation values ​​of each network layer of a floating-point model, inputting random data into the floating-point model to obtain the input activation values ​​of each network layer, performing statistical analysis on the input activation values ​​of each network layer based on the data distribution to obtain a data range of the input activation values ​​of each network layer, determining quantization parameters of the input activation values ​​of each network layer based on endpoint values ​​of the data range, and performing quantization processing on the input activation values ​​generated during the inference process of each network layer using the quantization parameters of the input activation values ​​of each network layer using the floating-point model.

[0007] In the scheme shown in the present disclosure, the user provides the data distribution of the input activation value, and uses random data as the input of the floating-point model, so as to determine the quantization parameters of the input activation value and quantize the input activation value of the floating-point model. Therefore, there is no need to train the data to generate the model, which can save resource consumption.

[0008] In an optional manner, for each network layer of the floating-point model, the endpoint value of the data range of the input activation value of the network layer is first used to determine the initial value of the quantization parameter of the input activation value of the network layer, and then the input activation value of the network layer is quantized using the initial value of the quantization parameter to obtain the quantized input activation value, and the quantization error of the input activation value of the network layer is determined based on the quantized input activation value and the input activation value of the network layer, and the initial value of the quantization parameter of the input activation value of the network layer is updated until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimum, and the value of the quantization parameter of the input activation value of the network layer is obtained. In this way, the value of the quantization parameter can be obtained using the quantization error as a constraint condition.

[0009] In an optional manner, when the quantization parameter includes a scaling factor and a quantization bias, the scaling factor and the quantization bias are updated in an alternating update manner, and the processing is as follows: when the scaling factor is an initial value, the quantization bias is updated until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization bias; when the quantization bias update is completed, the scaling factor is updated until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the scaling factor.

[0010] In an optional manner, the method can also quantize the weight values ​​in the floating-point model, and the processing is as follows: for each group of the floating-point model, determine the quantization parameter corresponding to the group, and based on the quantization parameter corresponding to the group, quantize the weight values ​​in the group to obtain the quantized value of the weight values ​​in the group, which can further reduce the storage resources occupied by the floating-point model and save computing resources. Moreover, by determining the quantization parameter in groups, the quantization parameter can be determined quickly.

[0011] In an optional manner, when the weight values ​​in the floating-point model are quantized, the rounding direction of the weight values ​​with large quantization errors is adjusted according to the grouping of the weight values, and the processing is as follows: determining the perturbation tensor corresponding to the quantization rounding of the weight values ​​to be adjusted in the group, the weight values ​​to be adjusted are the first K weight values ​​in the group arranged from large to small when the quantization errors are quantized, the perturbation tensor is used to adjust the rounding direction of the weight values ​​to be adjusted when quantized, and determining the ratio of the weight values ​​to be adjusted to the corresponding scaling factor; rounding the sum of the ratio and the perturbation tensor to obtain a rounding result, and in the case of asymmetric quantization, updating the sum of the rounding result and the quantization bias corresponding to the weight value to be adjusted to the quantized value of the weight value to be adjusted, and in the case of symmetric quantization, updating the rounding result to the quantized value of the weight value to be adjusted. In this way, the quantization error can be further reduced.

[0012] In an optional manner, the method further includes: grouping the weight values ​​in the floating-point model according to the channels to which the weight values ​​belong, wherein the weight values ​​in the same channel belong to one group.

[0013] In a second aspect, the present disclosure provides a device for post-quantization without data training, which has the function of implementing the above-mentioned first aspect. The device includes at least one module, and the at least one module is used to implement the method for post-quantization without data training provided in the above-mentioned first aspect.

[0014] In some embodiments, the modules in the device for post-quantization without data training are implemented by software, and the modules in the device are program modules. In other embodiments, the modules in the device are implemented by hardware or firmware.

[0015] In a third aspect, the present disclosure provides a computing device, comprising a processor and a memory, wherein: the memory stores computer instructions, and the processor executes the computer instructions to implement the method of the first aspect and its optional implementation methods.

[0016] In a fourth aspect, the present disclosure provides a computer-readable storage medium storing computer instructions. When the computer instructions in the computer-readable storage medium are executed by a computing device, the computing device executes the method of the first aspect and its optional implementation methods.

[0017] In a fifth aspect, the present disclosure provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the method of the above-mentioned first aspect and its optional implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of a system architecture provided by an exemplary embodiment of the present disclosure;

[0019] Figure 2 is a schematic diagram of a system architecture provided by an exemplary embodiment of the present disclosure;

[0020] Figure 3 is a schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present disclosure;

[0021] Figure 4 is a flowchart of a method for post-training quantization provided by an exemplary embodiment of the present disclosure;

[0022] Figure 5 is a schematic diagram of a process of quantizing input activation values ​​provided by an exemplary embodiment of the present disclosure;

[0023] Figure 6 It is a flowchart of optimizing the rounding method of weight values ​​provided by an exemplary embodiment of the present disclosure;

[0024] Figure 7 It is a structural diagram of a device for quantization after data-free training provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0026] Some terminology concepts involved in the embodiments of the present disclosure are explained below.

[0027] 1. Model quantization is the process of converting a floating-point model directly into a fixed-point integer calculation model. For example, convert a floating-point model with a data type of single-precision floating-point (FP32) or half-precision floating-point (FP16) into an integer model with a data type of 8-bit integer (int8). The quantized model can reduce the amount of model calculation, effectively improve the inference speed, and reduce memory usage, which can support the deployment of deep learning models on devices with small memory.

[0028] 2. Post-training quantization is a commonly used deep learning model compression method. It means converting the floating-point parameters in the model into fixed-point parameters after model training. During the conversion, the quantization parameters used need to be calibrated or optimized to ensure that the accuracy loss after model quantization meets the requirements.

[0029] 3. Data-free post-training quantization (data-free PTQ) refers to a method of directly quantizing the floating-point model after training without the user providing a standard data set.

[0030] The following describes the background of the embodiments of the present disclosure.

[0031] Deep learning has a wide range of applications and landing needs in many fields, such as language recognition, image processing, and autonomous driving. Some scenarios, such as autonomous driving, have strict requirements on the latency and memory usage of deep learning model inference, so it is necessary to quantize the deep learning model after training to obtain a deep learning model with low latency and low memory usage. When performing post-training quantization, the quantization parameters of the input activation values ​​in the deep learning model need to be calculated using a standard data set, but the standard data set generally involves user privacy and is often not available. Therefore, it may not be possible to use the standard data set to quantize the input activation values ​​in the deep learning model. Based on this, the method of post-data quantization without data training came into being. This method quantizes the model in scenarios where users do not provide data, and has a wider range of applications than ordinary model quantization methods.

[0032] The disclosed embodiment provides a data-free post-training quantization method, in which a floating-point model can be quantized after data-free training without the need for a standard data set or a training data generation model, thereby saving resources.

[0033] In the following, the technical solution provided by the embodiment of the present disclosure is described in detail according to the execution subject, system architecture, hardware structure of the device in the system architecture and method flow of the method after data-free training quantization.

[0034] The following describes the execution body of the method for quantization after data-free training.

[0035] The execution subject of the method for quantization after data-free training is a device for quantization after data-free training, referred to as a quantization device. Optionally, the quantization device is a hardware device, and the hardware device is a computing device such as a terminal device and a server. Optionally, the quantization device is a software device, such as a set of software programs running on the hardware device.

[0036] The following describes the system architecture provided by the embodiments of the present disclosure.

[0037] In an optional manner, the present disclosure embodiment provides a system architecture 100. Figure 1As shown, the system architecture 100 includes a terminal device 101 and a computing device 102. The terminal device 101 and the computing device 102 are connected via a wired or wireless network. The terminal device 101 is a device used by a user, such as a desktop computer, a laptop computer, or a tablet computer, and the terminal device 101 is used for the user to interact with the computing device 102. For example, the user can input data distribution, etc. through the terminal device 101. The computing device 102 can be a server, etc., and the computing device 102 is used to quantize the pre-trained floating-point model using the data distribution and return the quantized model to the terminal device 101.

[0038] In another optional manner, the embodiment of the present disclosure provides another system architecture 200. Figure 2 As shown, the system architecture 200 includes a terminal device 101 and a public cloud 201. The terminal device 101 is connected to the public cloud 201 via a wired or wireless network. The terminal device 101 is a device used by the user, and the terminal device 101 is used for the user to interact with the public cloud 201. The public cloud 201 is an entity that uses basic resources to provide cloud services to users in a cloud computing model. The public cloud 201 can also be considered as a cloud environment. The public cloud 201 includes a cloud data center, which includes a large number of basic resources owned by a cloud service provider. The large number of basic resources include computing resources, storage resources, and network resources. The computing resources included in the cloud data center can be a computing device cluster, and the computing device cluster includes at least one computing device 102. The computing device 102 can be a server, etc. When a user uses a cloud service, the user can input data distribution to the public cloud 201 through an API or an interactive interface (the interactive interface can be a graphical user interface (GUI)). The computing device in the public cloud 201 receives the pre-trained floating-point model and data distribution input by the user, uses the data distribution to quantize the pre-trained floating-point model, and returns the quantized model to the terminal device 101.

[0039] In the system architecture 200, the method of post-quantization without data training can be abstracted into a cloud service by the cloud service provider in the public cloud 201 and provided to users. After the user obtains the permission to use the cloud service, the user can use the cloud service to perform post-quantization without data training.

[0040] In another optional manner, the embodiment of the present disclosure further provides a system architecture. The system architecture includes a terminal device, which is a device used by a user. The terminal device executes the method for quantization after data-free training in the embodiment of the present disclosure.

[0041] The following is an introduction to the hardware structure of the device.

[0042] like Figure 3As shown, the device 300 is optionally implemented by a general bus architecture. The device 300 includes at least one processor 301, a communication bus 302, a memory 303, and at least one network interface 304. The device 300 is the terminal device 101 and the computing device 102 mentioned above.

[0043] The processor 301 is, for example, a general-purpose central processing unit (CPU), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the disclosed solution. For example, the processor 301 includes an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0044] The communication bus 302 is used to transmit information between the above components. The communication bus 302 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0045] The memory 303 is, for example, a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 303 is, for example, independent and connected to the processor 301 via the communication bus 302. The memory 303 can also be integrated with the processor 301.

[0046] Optionally, the memory 303 is used to store a pre-trained floating-point model. When the device 300 needs to use the floating-point model, the processor 301 accesses the memory 303 to obtain the floating-point model stored in the memory 303.

[0047] The network interface 304 uses any transceiver-like device for communicating with other devices or communication networks. The network interface 304 includes a wired network interface and may also include a wireless network interface. The wired network interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless network interface may be a wireless local area network (WLAN) interface, a cellular network network interface, or a combination thereof, etc.

[0048] In a specific implementation, as an example, the processor 301 may include one or more CPUs.

[0049] In a specific implementation, as an example, the device 300 may include multiple processors. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0050] In some embodiments, the memory 303 is used to store a program code 3031 for executing the data-free training post-quantization in the present disclosure, and the processor 301 executes the program code 3031 stored in the memory 303. That is, the device 300 can implement the data-free training post-quantization method provided in the method embodiment through the processor 301 and the program code 3031 in the memory 303.

[0051] Combine the following Figures 4 to 6 Describe the process of post-training quantization.

[0052] In post-training quantization, the quantized value of the weight value can be expressed as W q =W+ΔW, where W represents the value of the weight value before quantization and ΔW represents the difference between the weight value before and after quantization. The value of the input activation value after quantization is represented as A q =A+ΔA, where A represents the value of the input activation value before quantization, ΔA represents the difference between the input activation value before and after quantization, and the final calculated value after quantization is expressed as formula (1).

[0053] W q A q =WA+AΔW+WΔA+ΔWΔA(1)

[0054] During quantization, the goal is to minimize the quantization error, which is expressed as formula (2), or during quantization, the goal is to make the quantization error less than or equal to a certain value.

[0055]

[0056] In the embodiment of the present disclosure, the quantization error is taken as the optimization target. Combining formula (1) and formula (2), it can be known that the quantization error mainly comes from the input activation value quantization and the weight value quantization. Therefore, in the embodiment of the present disclosure, in view of the quantization requirements of the no-data scenario, the input activation value quantization and the weight value quantization are performed separately to minimize the quantization error, or to make it less than or equal to a certain value.

[0057] From the above description, we can see that post-training quantization includes two parts: input activation value quantization and weight value quantization. Figure 4, when the input activation value is quantized, the input of the post-training quantization includes a pre-trained floating-point model and data distribution. The pre-trained floating-point model is a deep learning model that has been trained. The data type of the parameters in the deep learning model is a floating-point type. The deep learning model includes deep learning models for scenes such as speech recognition, text recognition, image processing, or vehicle driving. The embodiments of the present disclosure are not limited. The data type in the pre-trained floating-point model is any floating-point type, for example, FP32, FP16, or FP64, etc. The floating-point model includes multiple network layers connected in sequence. For example, the network layer of the floating-point model may include a convolutional layer, a fully connected layer, and a pooling layer, etc. The network layers included in different floating-point models may be different. The multiple network layers included in the floating-point model correspond to the application scenarios of the floating-point model. For example, the floating-point model is a deep learning model for text recognition, and the multiple network layers include an embedding layer, a convolutional layer, and a fully connected layer. The data distribution is the distribution statistics of the input activation value. For example, the data distribution is a normal distribution, etc. When the weight value is quantized, the input of the post-training quantization includes a pre-trained floating-point model. There is no order between the input activation value quantization and the weight value quantization. The input activation value can be quantized first, or the weight value can be quantized first. The embodiment of the present disclosure does not limit this. Figure 4 It is shown in that the input activation value is quantized first, and then the weight value is quantized. After the input activation value is quantized and the weight value is quantized, the quantized model is output.

[0058] The following describes the process of input activation value quantization and the process of weight value quantization respectively.

[0059] 1. Quantization process of activation value: This quantization process includes the quantization process of input activation value and the process of optimizing the rounding method.

[0060] 1.1. Quantization process of input activation value, see Figure 5 Steps 501 to 505 in Figure 5 The process shown in the previous Figure 2 The system architecture shown is used as an example to illustrate. Figure 5 The computing device in is the computing device 102 mentioned above.

[0061] Step 501, obtaining the data distribution input by the user.

[0062] In this embodiment, the user inputs data distribution through the terminal device, and the computing device receives the data distribution sent by the terminal device. The data distribution is the distribution conforming to the input activation values ​​of each network layer of the floating-point model, that is, the input activation values ​​of each network layer conform to the data distribution, such as the data distribution is a normal distribution or a Poisson distribution.

[0063] In an optional manner, if the pre-trained floating-point model is obtained by training the computing device, the user may not provide the floating-point model, which is equivalent to providing the user with an input interface after the computing device trains the floating-point model, and the user uploads the data distribution of the input activation value through the input interface, and the computing device receives the data distribution. If the pre-trained floating-point model is not obtained by training the computing device, the computing device provides the user with an input interface, and the user uploads the pre-trained floating-point model and the data distribution of the input activation value through the input interface.

[0064] In an optional manner, the user also inputs a quantization bit number n through a terminal device, where the quantization bit number represents the number of bits of each input activation value after quantization, and the computing device obtains the quantization bit number n.

[0065] Step 502: input random data into the floating point model to obtain input activation values ​​of each network layer.

[0066] In this embodiment, random data is stored in the computing device, and the random data is commonly used data that can make the input activation value of each network layer conform to the data distribution input by the user. Alternatively, random data is stored in the computing device, and the random data corresponds to a data distribution. The computing device obtains random data that conforms to the data distribution input by the user from the stored random data. Alternatively, the computing device generates random data that conforms to the data distribution based on the data distribution. The computing device inputs the random data into the floating-point model and performs forward reasoning to obtain the input activation value of each network layer of the floating-point model. For the network layer that is the frontmost among the multiple network layers connected in sequence in the floating-point model, the input activation value of the network layer can be a feature of the random data. For any network layer except the frontmost network layer among the multiple network layers, the input activation value of the network layer is the feature data output by the previous network layer before the network layer.

[0067] Step 503: Based on the data distribution, statistical analysis is performed on the input activation values ​​of each network layer to obtain the data range of the input activation values ​​of each network layer.

[0068] In this embodiment, the data distribution input by the user is the data distribution that the input activation value of each network layer conforms to. For any network layer, the computing device uses the data distribution to determine the maximum and minimum values ​​of the input activation value of the network layer, and obtains the data range of the input activation value. For example, the data distribution is a normal distribution. For the kth network layer of the floating-point model, the input activation value of the kth network layer can be used to calculate the parameters μ and σ of the normal distribution, where μ is the mathematical expectation and σ is the standard deviation. In the normal distribution, the probability that the data falls in (μ-3σ, μ+3σ) is 99.74%, so (μ-3σ, μ+3σ) can be used as the data range of the input activation value of the kth network layer, where μ-3σ is the minimum value of the data range and μ+3σ is the maximum value of the data range.

[0069] Given the number of quantization bits n, the range of the quantized input activation value is [-2 n-1 , 2 n-1 -1].

[0070] Step 504: Determine the quantization parameters of the input activation values ​​of each network layer based on the endpoint values ​​of the data range.

[0071] In this embodiment, for any network layer, the endpoint values ​​of the data range of the network layer are the maximum value and the minimum value of the input activation value of the network layer, and the computing device uses the maximum value and the minimum value to determine the quantization parameter of the input activation value of the network layer. In the case of asymmetric quantization, the quantization parameter includes a scaling factor and a quantization bias, and in the case of symmetric quantization, the quantization parameter includes a scaling factor.

[0072] In an optional manner, the process of determining the quantization parameter using the endpoint values ​​of the data range is as follows:

[0073] In the case of asymmetric quantization, the quantization parameters include a scaling factor and a quantization bias. For any network layer, the computing device uses formula (3) to calculate the initial value of the scaling factor corresponding to the network layer, and uses formula (4) to calculate the initial value of the quantization bias corresponding to the network layer.

[0074]

[0075]

[0076] In formula (3) and formula (4), scale1 represents a scaling factor, max_val represents the maximum value of the input activation value of the network layer, min_val represents the minimum value of the input activation value of the network layer, n represents the number of quantization bits, offset1 represents the quantization offset, and round represents a rounding operation, which can be a rounding operation, an upward rounding operation, or a downward rounding operation.

[0077] Then, the computing device uses the initial value of the quantization parameter and formula (5) to quantize the input activation value of the network layer to obtain the quantized value of the input activation value, dequantizes the quantized value of the input activation value to obtain the dequantized value, and uses formula (6) or (7) to calculate the quantization error of the input activation value of the network layer. It is determined whether the quantization error is less than or equal to the target value corresponding to the network layer. If the quantization error is less than or equal to the target value, the initial value of the scaling factor is determined as the value of the scaling factor, and the initial value of the quantization bias is determined as the value of the quantization bias. If the quantization error is greater than the target value, the initial value of the quantization parameter of the input activation value of the network layer is updated, and the above process is continued until the quantization error of the input activation value of the network layer is less than or equal to the target value, and the value of the quantization parameter of the input activation value of the network layer is obtained. Alternatively, the initial value of the quantization parameter of the input activation value of the network layer is updated until the quantization error of the input activation value of the network layer is minimized, and the value of the quantization parameter of the input activation value of the network layer is obtained. Among them, the target value can be set based on the actual situation. The larger the number of input activation values ​​of the network layer, the larger the target value is set, and the smaller the number of input activation values ​​of the network layer, the smaller the set target value is.

[0078]

[0079] In formula (5), A2 represents the value of the input activation value after quantization, A1 represents the value of the input activation value before quantization, scale1 represents the scaling factor, offset1 represents the quantization offset, and round represents the rounding operation.

[0080]

[0081]

[0082] In formula (6), L(A1, A2) represents the quantization error, A 1i represents the value of the i-th input activation value before quantization, i is an integer less than or equal to m, A 2i It represents the dequantized value of the quantized value of the i-th input activation value. The dequantized value is a floating point number. m represents the number of input activation values ​​of the network layer.

[0083] Optionally, in the case of asymmetric quantization, the computing device may use an alternating update method to determine the scaling factor and the quantization bias. For example, for any network layer, when the scaling factor is an initial value, the input activation value of the network layer is quantized using the initial value of the scaling factor and the initial value of the quantization bias to obtain the quantized value of the input activation value of the network layer, see formula (5).

[0084] Then the computing device dequantizes the quantized value and uses formula (6) or (7) to calculate the quantization error corresponding to the network layer. The computing device increases the initial value of the quantization bias by the first adjustment amplitude, substitutes the initial value of the scaling factor and the adjusted quantization bias into formula (5), quantizes the input activation value of the network layer, obtains the quantized value of the input activation value of the network layer, and uses formula (6) or (7) to calculate the quantization error corresponding to the network layer. The computing device determines whether the quantization error meets the minimum or is less than or equal to the target value. If not, the quantization bias is adjusted continuously, and so on, until the value of the quantization bias is obtained when the quantization error is minimum, or until the value of the quantization bias is obtained when the quantization error is less than or equal to the target value.

[0085] Then, after obtaining the quantization bias, the quantization bias is fixed and the scaling factor is updated to determine the value of the scaling factor when the quantization error is minimum, or the value of the scaling factor when the quantization error is less than or equal to the target value. The process of updating the scaling factor is similar to the process of updating the quantization bias, which will not be repeated here.

[0086] In this way, after multiple alternating updates of the quantization bias and the scaling factor, the final scaling factor and quantization bias are obtained.

[0087] In the case of symmetric quantization, the target quantization parameter includes a scaling factor. For any network layer, the computing device uses formula (3) to calculate the initial value of the scaling factor corresponding to the network layer. For any network layer, only the scaling factor corresponding to the network layer is adjusted, which is equivalent to the quantization bias value being 0. The process of determining the scaling factor refers to the processing of asymmetric quantization, which will not be repeated here.

[0088] Step 505, during the inference process of the floating-point model, the quantization parameters of the input activation values ​​of each network layer are used to quantize the input activation values ​​generated during the inference process of each network layer.

[0089] In this embodiment, for each network layer, the quantization parameters of all input activation values ​​of the network layer are the same. In the subsequent inference process using the floating-point model, after the inference data is input into the floating-point model, the input activation value of each network layer is obtained, and the input activation value of the network layer is quantized using the quantization parameter of the input activation value of the network layer, and then the inference calculation is performed. In this way, the floating-point input activation value is converted into an integer input activation value, which can reduce the amount of calculation.

[0090] The device that performs step 505 may be the same device as the device that performs steps 501 to 504, or may be a different device. For example, the computing device determines the quantization parameter of the input activation value and performs the inference process. Figure 5After the process shown obtains the quantization parameter of the input activation value, the computing device returns the quantization parameter to the terminal device, and the terminal device stores the quantization parameter. Later in the inference process, the terminal device uses the quantization parameter to quantize the input activation value, which can reduce the amount of calculation of the input activation value.

[0091] use Figure 5 The process shown does not require users to provide data when quantizing the model, which can protect user privacy and has a wider range of applications. In addition, there is no need for additional training to obtain a data generator, which can save resources and improve quantization efficiency.

[0092] In the scenario of post-quantization without data training, the quantization parameters of the input activation values ​​are initialized according to the data distribution of the input activation values, which has higher accuracy than random estimation. In addition, in the scenario of post-quantization without data training, there is no requirement for the structure of the floating-point model, and the generalization ability is stronger.

[0093] 1.2. Optimization process of rounding method during input activation value quantization.

[0094] After the quantization parameters are determined, for the input activation values ​​of any network layer, when no perturbation tensor is introduced, the input activation values ​​of the network layer are quantized and rounded to obtain the quantized value of the input activation value (quantization is performed using formula (5), and in the case of symmetric quantization, the quantization bias is 0), and the quantization error of the input activation value of the network layer is calculated using formula (6) or (7).

[0095] Then determine the K input activation values ​​in the network layer that are ranked top in quantization error from large to small, introduce a perturbation tensor for the K input activation values ​​(see formula (8) for asymmetric quantization and formula (9) for symmetric quantization), quantize and round the input activation values ​​of the network layer to obtain the quantized value of the input activation value, continue to use formula (6) or (7) to calculate the quantization error of the input activation value to be adjusted of the network layer (m represents the number of input activation values ​​to be adjusted), update the perturbation tensor until the quantization error is minimized, and obtain the perturbation tensor with the minimum quantization error. Alternatively, calculate whether the quantization error of each input activation value to be adjusted is the minimum. If it is the minimum, obtain the perturbation tensor corresponding to the input activation value to be adjusted. If it is not the minimum, update the perturbation tensor until the quantization error of each input activation value to be adjusted is the minimum, and obtain the perturbation tensor with the minimum quantization error. The perturbation tensors corresponding to the K input activation values ​​of the network layer can be the same, partially the same, or completely different.

[0096]

[0097]

[0098] Wherein, in formulas (8) and (9), Δ1 is the disturbance tensor.

[0099] In addition, the principle of introducing the perturbation tensor is: assuming that the rounding operation is rounding, the decimal part of the original ratio of the input activation value to the scaling factor is less than 5. By introducing the perturbation tensor, the decimal part is changed to be greater than or equal to 5, thereby changing the rounded value; or, the decimal part of the original ratio of the input activation value to the scaling factor is greater than or equal to 5. By introducing the perturbation tensor, the decimal part is changed to be less than 5, thereby changing the rounded value.

[0100] It should be noted that after determining the quantization parameters, the perturbation tensor is introduced for the input activation values ​​with relatively large quantization errors in each network layer, and the rounding direction is further adjusted to further reduce the quantization error.

[0101] 2. The process of quantizing weight values: The process of quantizing weight values ​​includes the process of determining the quantization process of weight values ​​and the process of optimizing the rounding method.

[0102] 2.1. The quantization process of weight value is:

[0103] The weight values ​​in the floating point model are divided into a plurality of groups, each group including one or more weight values. For the weight values ​​in each group of the floating point model, the quantization parameters corresponding to the weight values ​​in the group are the same.

[0104] In an optional manner, in the case of asymmetric quantization, the quantization parameter corresponding to the weight value in the group is determined according to formula (10), and in the case of symmetric quantization, the quantization parameter corresponding to the weight value in the group is determined according to formula (11).

[0105]

[0106]

[0107] Wherein, L(W1,W2) represents the quantization error of the weight value in a group, and the expression is shown in formula (12). In formula (10), When minimizing the quantization error, the values ​​of the scaling factor scale2 and the quantization bias offset2 are determined by W2 using formula (13). In formula (11), When minimizing the quantization error, the value of scale2 and W2 are determined using formula (14).

[0108]

[0109]

[0110]

[0111]

[0112] In formula (12) and formula (15), W1 represents the value of the weight value before quantization, W2 represents the value of the weight value after quantization and the value after dequantization, W 1j represents the value of the jth weight value before quantization, j is an integer less than or equal to g, W 2j represents the dequantized value of the jth weight value after quantization, and g represents the number of weight values ​​in the group. In formula (13) and formula (14), scale2 represents the scaling factor of the weight value, offset2 represents the quantization bias of the weight value, round represents the rounding operation, which can be rounding, rounding up, or rounding down, and Δ2 represents the perturbation tensor in the following text, which is 0 here.

[0113] In formula (10), two parameters, the scaling factor scale2 and the quantization offset offset2, need to be adjusted. When adjusting the two parameters, the initial values ​​of the two parameters are preset values. Under the premise of fixing scale2, adjust offset2 so that the quantization error L(W1, W2) of the weight value in the group is minimized, and then fix offset2 to offset2 when L(W1, W2) is minimized, and adjust scale2 so that L(W1, W2) is minimized, until the number of times the process is executed reaches the preset number of times, and the scale2 and offset2 when the process is executed for the last time are determined as the final quantization parameters. The preset number is set according to an empirical value, such as 10.

[0114] In another optional manner, when adjusting the quantization parameter of the weight value, when L(W1, W2) is less than or equal to the target value, it can also be considered that the quantization parameter is determined to be obtained.

[0115] For the weight value in each group, after the quantization parameter is determined, a quantization process is performed using the quantization parameter to obtain a quantized value of the weight value.

[0116] 2.2. The process of rounding method optimization and quantification based on the optimized rounding method.

[0117] After determining the quantization parameters of the weight values ​​in each group, the quantization errors of some weight values ​​in each group may be relatively large. In order to further reduce the quantization errors of these weight values, a quantization process based on the optimized rounding method is performed. Figure 6 Steps 601 to 603 in Figure 6 The process shown in the previous Figure 2 The system architecture shown is used as an example to illustrate. Figure 6 The computing device in is the computing device 102 mentioned above.

[0118] Step 601: for each weight value in the floating-point model, determine the disturbance tensor corresponding to the quantization and rounding of the weight value to be adjusted in the group.

[0119] In this embodiment, for the weight value of any group, when the perturbation tensor is not introduced, in the case of asymmetric quantization, the quantized value of the weight value is calculated using formula (13) (Δ2 is 0), and in the case of symmetric quantization, the quantized value of the weight value is calculated using formula (14) (Δ2 is 0). The weight value in the group is quantized and rounded to obtain the quantized value of the weight value, and the quantization error of the weight value in the group is calculated using formula (12) or (15).

[0120] Then, the quantization errors of the weight values ​​in the group are sorted from large to small, and the weight values ​​of the top K quantization errors are determined. The weight values ​​of the K quantization errors are the weight values ​​to be adjusted. For different groups, the value of K may be different. The more weight values ​​there are in the group, the larger the value of K. Conversely, the fewer weight values ​​there are in the group, the smaller the value of K.

[0121] Then, a perturbation tensor is introduced for the weight values ​​to be adjusted in the group. In the case of asymmetric quantization, formula (13) is used to calculate the quantized value of the weight value to be adjusted. In the case of symmetric quantization, formula (14) is used to calculate the quantized value of the weight value to be adjusted. Continue to use formula (12) or (15) to calculate the quantization error of the weight value to be adjusted (g represents the number of weight values ​​to be adjusted) until a suitable perturbation tensor is selected to minimize the quantization error. This process is represented by formula (16). Alternatively, calculate whether the quantization error of each weight value to be adjusted is the smallest. If it is the smallest, obtain the perturbation tensor corresponding to the weight value to be adjusted. If it is not the smallest, update the perturbation tensor until the quantization error of each weight value to be adjusted is the smallest, and obtain the perturbation tensor when the quantization error is the smallest. For any group, the perturbation tensors corresponding to the weight values ​​to be adjusted in the group can be the same, partially the same, or completely different.

[0122]

[0123] In formula (16), The value of Δ2 when minimizing the quantization error, the expression of L(W1, W2) can be found in formula (12) or (15).

[0124] It should be noted that the principle of introducing the perturbation tensor is: assuming that the rounding operation is rounding, the decimal part of the original ratio of the weight value to the scaling factor is less than 5. By introducing the perturbation tensor, the decimal part is changed to greater than or equal to 5, thereby changing the rounded value; or, the decimal part of the original ratio of the weight value to the scaling factor is greater than or equal to 5. By introducing the perturbation tensor, the decimal part is changed to less than 5, thereby changing the rounded value.

[0125] Step 602: Determine the ratio of the weight value to be adjusted to the corresponding scaling factor.

[0126] Step 603, rounding the ratio and the sum of the disturbance tensor to obtain a rounding result. In the case of asymmetric quantization, the sum of the rounding result and the quantization bias corresponding to the weight value to be adjusted is updated to the quantized value of the weight value to be adjusted. In the case of symmetric quantization, the rounding result is updated to the quantized value of the weight value to be adjusted.

[0127] In this embodiment, in the case of asymmetric quantization, formula (13) is used to calculate the quantized value of the weight value to be adjusted, and in the case of symmetric quantization, formula (14) is used to calculate the quantized value of the weight value to be adjusted.

[0128] In formulas (13) and (14), Indicates the ratio of the weight value to be adjusted to the scaling factor.

[0129] For any group of weights to be adjusted, you can use Figure 6 The process shown quantizes the weight values ​​to be adjusted.

[0130] In an optional manner, there are multiple ways to group weight values ​​in a floating-point model, and two feasible ways are provided below.

[0131] Method 1: The weight values ​​in the floating-point model are grouped according to the channels to which the weight values ​​belong, and the weight values ​​belonging to the same channel are grouped into the same group.

[0132] For example, the weight value of the convolution layer of the floating-point model is a four-dimensional tensor, which is expressed as s×e×h×r, where s represents the number of output channels of the convolution layer, e represents the number of input channels of the convolution layer, and h×r represents the h×r convolution kernel. During quantization, e×h×r weight values ​​are a group.

[0133] Method 2: The weight values ​​in the floating-point model are grouped according to the network layer to which the weight values ​​belong, and the weight values ​​belonging to the same network layer are grouped into the same group.

[0134] In the disclosed embodiment, when quantizing the weight value, the quantization error is minimized or less than or equal to the target value as a constraint condition. Quantizing the weight value can reduce the error caused by quantization and improve the inference accuracy of the quantized model.

[0135] In the embodiments of the present disclosure, the products that may be involved include model compression tools or reasoning model deployment tools. When determining whether infringement occurs later, the following methods can be used to determine whether infringement occurs:

[0136] 1. Check whether input data is required from the quantized input after training and whether the data distribution of the input activation value is required.

[0137] 2. From the model compression process and log analysis, whether the quantization process uses the data set, whether additional data processing steps are performed, and whether it is necessary to initialize different random data for different data distributions to calculate the quantization parameters of the input activation values.

[0138] 3. Check the functions released by the model compression tool to see whether the above-mentioned process of quantization after data-free training is integrated.

[0139] Figure 7 1 is a structural diagram of a device for quantization after data-free training provided by an embodiment of the present disclosure. The device can be implemented as part or all of the device through software, hardware, or a combination of both. The device provided by an embodiment of the present disclosure can implement the embodiment of the present disclosure. Figures 4 to 6 The process described above, the device includes: an acquisition module 710 and a quantification module 720, wherein:

[0140] An acquisition module 710 is used to acquire data distribution of user input, where the data distribution is the distribution conforming to the input activation values ​​of each network layer of the floating-point model, and can be specifically used to implement the acquisition function of step 501 and execute the implicit steps included in step 501;

[0141] The quantization module 720 is used to:

[0142] Inputting random data into the floating point model to obtain input activation values ​​of each network layer;

[0143] Based on the data distribution, statistically analyzing the input activation values ​​of each network layer to obtain the data range of the input activation values ​​of each network layer;

[0144] Determining quantization parameters of input activation values ​​of each network layer based on endpoint values ​​of the data range;

[0145] During the inference process using the floating-point model, the quantization parameters of the input activation values ​​of each network layer are used to quantize the input activation values ​​generated during the inference process of each network layer, which can be specifically used to implement the input activation value quantization function of steps 502 to 505 and execute the implicit steps contained in steps 502 to 505.

[0146] In an optional manner, the quantization module 720 is used to:

[0147] For each network layer of the floating-point model, using the endpoint values ​​of the data range of the input activation values ​​of the network layer, determining the initial value of the quantization parameter of the input activation value of the network layer;

[0148] Using the initial value of the quantization parameter, quantizing the input activation value of the network layer to obtain a quantized input activation value;

[0149] Determining a quantization error of the input activation value of the network layer based on the quantized input activation value and the input activation value of the network layer;

[0150] Update the initial value of the quantization parameter of the input activation value of the network layer until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization parameter of the input activation value of the network layer.

[0151] In an optional manner, the quantization parameter includes a scaling factor and a quantization offset;

[0152] The quantization module 720 is used to:

[0153] When the scaling factor is an initial value, updating the quantization bias until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization bias;

[0154] When the quantization bias update is completed, the scaling factor is updated until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the scaling factor.

[0155] In an optional manner, the quantization module 720 is further configured to:

[0156] For each group of the floating-point model, determine a quantization parameter corresponding to the group, wherein in the case of asymmetric quantization, the quantization parameter corresponding to the group includes a scaling factor and a quantization bias, and in the case of symmetric quantization, the quantization parameter corresponding to the group includes a scaling factor;

[0157] Based on the quantization parameter corresponding to the group, the weight value in the group is quantized to obtain the quantized value of the weight value in the group.

[0158] In an optional manner, the quantization module 720 is further configured to:

[0159] Determine a disturbance tensor corresponding to quantization rounding of the weight value to be adjusted in the group, wherein the weight value to be adjusted is the first K weight values ​​arranged from large to small in the group when quantization errors are quantized, and the disturbance tensor is used to adjust the rounding direction of the weight value to be adjusted when quantization rounding;

[0160] Determining a ratio of the weight value to be adjusted to a corresponding scaling factor;

[0161] The ratio and the sum of the disturbance tensor are rounded to obtain a rounded result. In the case of asymmetric quantization, the sum of the rounded result and the quantization bias corresponding to the weight value to be adjusted is updated to the quantized value of the weight value to be adjusted. In the case of symmetric quantization, the rounded result is updated to the quantized value of the weight value to be adjusted.

[0162] In an optional manner, the acquisition module 710 is used to:

[0163] The weight values ​​in the floating-point model are grouped according to the channels to which the weight values ​​belong, wherein the weight values ​​in the same channel belong to one group.

[0164] Figure 7 For the detailed process of quantizing the input activation value of the device for quantization after data-free training shown, please refer to the description in the previous embodiments, which will not be repeated here. Figure 7 The device shown in the figure is quantized after training without data. Figure 1 and attached Figure 2 The computing device 102 in FIG.

[0165] In an optional manner, a computer program product is provided, the computer program product comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computing device executes Figures 4 to 6 The process shown.

[0166] Those of ordinary skill in the art will appreciate that the various method steps and units described in the embodiments disclosed in this disclosure can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0167] In the several embodiments provided in the present disclosure, it should be understood that the disclosed system architecture, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.

[0168] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0169] In addition, each module in each embodiment of the present disclosure may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or software modules.

[0170] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0171] The term "plurality" in the present disclosure means two or more.

[0172] The above description is only an exemplary embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for quantization after data-free training, characterized in that: The method comprises: Obtaining data distribution of user input, where the data distribution is a distribution conforming to input activation values ​​of each network layer of the floating-point model; Inputting random data into the floating point model to obtain input activation values ​​of each network layer; Based on the data distribution, statistically analyzing the input activation values ​​of each network layer to obtain the data range of the input activation values ​​of each network layer; Determining quantization parameters of input activation values ​​of each network layer based on endpoint values ​​of the data range; During the inference process using the floating-point model, the input activation values ​​generated during the inference process of each network layer are quantized using the quantization parameters of the input activation values ​​of each network layer.

2. The method according to claim 1, characterized in that The step of determining the quantization parameters of the input activation values ​​of each network layer based on the endpoint values ​​of the data range includes: For each network layer of the floating-point model, using the endpoint values ​​of the data range of the input activation values ​​of the network layer, determining the initial value of the quantization parameter of the input activation value of the network layer; Using the initial value of the quantization parameter, quantizing the input activation value of the network layer to obtain a quantized input activation value; Determining a quantization error of the input activation value of the network layer based on the quantized input activation value and the input activation value of the network layer; Update the initial value of the quantization parameter of the input activation value of the network layer until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization parameter of the input activation value of the network layer.

3. The method according to claim 2, characterized in that The quantization parameters include a scaling factor and a quantization bias; The updating of the initial value of the quantization parameter of the input activation value of the network layer until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization parameter of the input activation value of the network layer, includes: When the scaling factor is an initial value, updating the quantization bias until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization bias; When the quantization bias update is completed, the scaling factor is updated until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the scaling factor.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: For each group of the floating-point model, determine a quantization parameter corresponding to the group, wherein in the case of asymmetric quantization, the quantization parameter corresponding to the group includes a scaling factor and a quantization bias, and in the case of symmetric quantization, the quantization parameter corresponding to the group includes a scaling factor; Based on the quantization parameter corresponding to the group, the weight value in the group is quantized to obtain the quantized value of the weight value in the group.

5. The method according to claim 4, characterized in that The method further comprises: Determine a disturbance tensor corresponding to quantization rounding of the weight value to be adjusted in the group, wherein the weight value to be adjusted is the first K weight values ​​arranged from large to small in the group when quantization errors are quantized, and the disturbance tensor is used to adjust the rounding direction of the weight value to be adjusted when quantization rounding; Determining a ratio of the weight value to be adjusted to a corresponding scaling factor; The ratio and the sum of the disturbance tensor are rounded to obtain a rounded result. In the case of asymmetric quantization, the sum of the rounded result and the quantization bias corresponding to the weight value to be adjusted is updated to the quantized value of the weight value to be adjusted. In the case of symmetric quantization, the rounded result is updated to the quantized value of the weight value to be adjusted.

6. The method according to claim 4 or 5, characterized in that: The method further comprises: The weight values ​​in the floating-point model are grouped according to the channels to which the weight values ​​belong, wherein the weight values ​​in the same channel belong to one group.

7. A device for quantization after data-free training, characterized in that: The device comprises: An acquisition module, used to acquire data distribution of user input, where the data distribution is the distribution conforming to the input activation values ​​of each network layer of the floating-point model; Quantization module for: Inputting random data into the floating point model to obtain input activation values ​​of each network layer; Based on the data distribution, statistically analyzing the input activation values ​​of each network layer to obtain the data range of the input activation values ​​of each network layer; Determining quantization parameters of input activation values ​​of each network layer based on endpoint values ​​of the data range; During the inference process using the floating-point model, the input activation values ​​generated during the inference process of each network layer are quantized using the quantization parameters of the input activation values ​​of each network layer.

8. The device according to claim 7, characterized in that The quantization module is used for: For each network layer of the floating-point model, using the endpoint values ​​of the data range of the input activation values ​​of the network layer, determining the initial value of the quantization parameter of the input activation value of the network layer; Using the initial value of the quantization parameter, quantizing the input activation value of the network layer to obtain a quantized input activation value; Determining a quantization error of the input activation value of the network layer based on the quantized input activation value and the input activation value of the network layer; Update the initial value of the quantization parameter of the input activation value of the network layer until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization parameter of the input activation value of the network layer.

9. The device according to claim 8, characterized in that The quantization parameters include a scaling factor and a quantization bias; The quantization module is used for: When the scaling factor is an initial value, updating the quantization bias until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the quantization bias; When the quantization bias update is completed, the scaling factor is updated until the quantization error of the input activation value of the network layer is less than or equal to the target value, or is minimized, so as to obtain the value of the scaling factor.

10. The device according to any one of claims 7 to 9, characterized in that The quantization module is further used for: For each group of the floating-point model, determine a quantization parameter corresponding to the group, wherein in the case of asymmetric quantization, the quantization parameter corresponding to the group includes a scaling factor and a quantization bias, and in the case of symmetric quantization, the quantization parameter corresponding to the group includes a scaling factor; Based on the quantization parameter corresponding to the group, the weight value in the group is quantized to obtain the quantized value of the weight value in the group.

11. The device according to claim 10, characterized in that The quantization module is further used for: Determine a disturbance tensor corresponding to quantization rounding of the weight value to be adjusted in the group, wherein the weight value to be adjusted is the first K weight values ​​arranged from large to small in the group when quantization errors are quantized, and the disturbance tensor is used to adjust the rounding direction of the weight value to be adjusted when quantization rounding; Determining a ratio of the weight value to be adjusted to a corresponding scaling factor; The ratio and the sum of the disturbance tensor are rounded to obtain a rounded result. In the case of asymmetric quantization, the sum of the rounded result and the quantization bias corresponding to the weight value to be adjusted is updated to the quantized value of the weight value to be adjusted. In the case of symmetric quantization, the rounded result is updated to the quantized value of the weight value to be adjusted.

12. The device according to claim 10 or 11, characterized in that The acquisition module is used to: The weight values ​​in the floating-point model are grouped according to the channels to which the weight values ​​belong, wherein the weight values ​​in the same channel belong to one group.

13. A computing device, characterized in that: The computing device comprises a processor and a memory, wherein: The memory stores computer instructions; The processor executes the computer instructions to cause the computing device to perform the method of any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions. When the computer instructions in the computer-readable storage medium are executed by a computing device, the computing device executes the method according to any one of claims 1 to 6.

Images