Neural network model quantification method and related equipment

Abstract

The embodiment of the invention provides a neural network model quantification method and related equipment, and relates to the field of artificial intelligence. In the method, a neural network model is firstly obtained; and reasoning the neural network model based on the test set to obtain an activation value tensor and / or a weight tensor of a quantifiable operator in the neural network model. And obtaining a quantitative model of the neural network model based on the activation value tensor and / or the weight tensor of the quantizable operator. In the scheme, reasoning is performed on the neural network based on the test set to obtain the activation value tensor and / or the weight tensor of the quantizable operator, and activation value quantization processing and / or weight quantization processing are / is performed on the neural network model based on the activation value tensor and / or the weight tensor of the quantizable operator to obtain the quantization model corresponding to the neural network model. The quantification method of the scheme can be automatically executed by the first device for deploying the neural network model, and the quantification process does not depend on a quantification toolkit, so that the usability is high.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a neural network model quantization method and related equipment. Background Art

[0002] With the continuous development of artificial intelligence (AI), more and more end-side devices are embedded with AI capabilities, and deep neural network models, as the core of AI algorithms, have also seen vigorous development. By quantifying deep neural network models, the following benefits can be obtained on the end side:

[0003] 1. Reduce the disk space occupied when storing models. Usually, in the development stage of neural network (NN) models, 32-bit floating point numbers are used to store model parameters. However, through model quantization technology, 32-bit floating point numbers can be converted to 8-bit or lower-bit fixed-point numbers for storage. In this way, the disk space occupied by model parameters will be reduced by 4 times.

[0004] 2. It shortens the reasoning time of the neural network model. Under the same number of floating-point operations (FLOPs), since the quantized model uses fixed-point number reasoning, 8-bit fixed-point number technology is usually faster than 32-bit floating-point number calculation.

[0005] 3. Achieve lower power consumption. The number of bits processed during the calculation of the quantized model is smaller, so the power consumption is lower.

[0006] Due to the above benefits, quantization technology is used in terminal applications. Quantization technology includes offline quantization and dynamic quantization. Offline quantization refers to the conversion of floating-point models to vectorized models on the development server and the calculation of quantization-related parameters on the server before the model is deployed to the terminal device. However, with offline quantization, the inference calculation of the neural network model needs to rely on the third-party framework of machine learning (such as tensorflow, pytroch, etc.). In the quantization process, the quantization parameters of the statistical model must rely on the third-party framework, which means that developers need to rely on the quantization toolkit, need to be aware of the entire quantization process, and need to actively complete the quantization.

[0007] Therefore, it is necessary to provide a new quantification method. Summary of the invention

[0008] The present application provides a neural network model quantization method and related equipment. The method can be automatically executed by a first device that deploys the neural network model, and the quantization process does not rely on a quantization toolkit.

[0009] In a first aspect, a neural network model quantization method is provided, which can be executed by a first device or a chip in the first device. The first device is a device for deploying a neural network model.

[0010] The neural network model quantization method comprises the following steps: obtaining a neural network model. Inferring the neural network model based on a test set to obtain an activation value tensor and / or a weight tensor of a quantizable operator in the neural network model. Obtaining a quantized model of the neural network model based on the activation value tensor and / or the weight tensor of the quantizable operator. The neural network model is a floating point neural network model.

[0011] It can be seen that in this solution, the neural network is inferred based on the test set to obtain the activation value tensor and / or weight tensor of the quantizable operator, and the neural network model is subjected to activation value quantization processing and / or weight quantization processing based on the activation value tensor and / or weight tensor of the quantizable operator to obtain a quantized model corresponding to the neural network model. The quantization method of this solution can be automatically executed by the first device that deploys the neural network model, and the quantization process does not rely on the quantization toolkit, and is easy to use.

[0012] In a possible implementation of the first aspect, the test set includes test data and labels corresponding to the test data; and the neural network model quantization method further includes the following steps: processing the test data in the test set by the quantization model to obtain a processing result. The accuracy of the quantization model is determined based on the processing result and the labels corresponding to the test data.

[0013] In this scheme, the quantization model is used to process the test data to obtain the corresponding processing results, and then the accuracy of the quantization model, that is, the level of quantization accuracy, can be determined based on the processing results and the labels corresponding to the test data. The quantization accuracy indicates the degree of closeness between the processing results and the labels of the test data. For example, the more the number of identical processing results of the test data in the test set and the labels corresponding to the test data is, the higher the quantization accuracy is.

[0014] In a possible implementation of the first aspect, the neural network model quantization method further includes: when the accuracy of the quantization model does not meet the accuracy requirement, obtaining a new test set. When the accuracy of the quantization model does not meet the accuracy requirement, obtaining a new test set to generate a new quantization model.

[0015] In a possible implementation of the first aspect, the above-mentioned acquisition of the quantization model of the neural network model based on the activation value tensor and / or weight tensor of the quantizable operator specifically includes the following steps: acquiring the quantization key parameters of the neural network model based on the activation value tensor and / or weight tensor of the quantizable operator. Generating the quantization model based on the quantization key parameters and the neural network model.

[0016] In a possible implementation manner of the first aspect, the quantization key parameters include a scale factor and a zero point.

[0017] In a possible implementation of the first aspect, the above-mentioned neural network model quantization method is applied to the debugging stage of the neural network model.

[0018] In a second aspect, the present application further provides a first device, which is a device for deploying a neural network model. The first device includes an acquisition module and an inference module, wherein:

[0019] The acquisition module is used to obtain the neural network model.

[0020] The inference module is used to perform inference on the neural network model based on the test set to obtain the activation value tensor and / or weight tensor of the quantizable operator in the neural network model.

[0021] The acquisition module is also used to acquire a quantized model of the neural network model based on the activation value tensor and / or weight tensor of the quantizable operator.

[0022] The above neural network model is a floating-point neural network model.

[0023] In this solution, the first device infers the neural network based on the test set to obtain the activation value tensor and / or weight tensor of the quantizable operator, and performs activation value quantization processing and / or weight quantization processing on the neural network model based on the activation value tensor and / or weight tensor of the quantizable operator to obtain a quantization model corresponding to the neural network model. In this application, the quantization method of this solution can be executed by the first device that deploys the neural network model to achieve automatic quantization, and the quantization process does not rely on the quantization toolkit, and is easy to use.

[0024] In a possible implementation manner of the second aspect, the test set includes test data and labels corresponding to the test data; the first device further includes a processing module and a determination module, wherein:

[0025] The processing module is used to process the test data in the test set through the quantization model to obtain a processing result.

[0026] The determination module is used to determine the accuracy of the quantization model based on the processing results and the labels corresponding to the test data.

[0027] In a possible implementation of the second aspect, the acquisition module is further used to acquire a new test set when the accuracy of the quantization model does not meet the accuracy condition.

[0028] In a possible implementation of the second aspect, the acquisition module is specifically used to:

[0029] The key quantization parameters of the neural network model are obtained based on the activation value tensor and / or weight tensor of the quantizable operator.

[0030] Generate a quantization model based on quantization key parameters and neural network model.

[0031] In a possible implementation manner of the second aspect, the above-mentioned key quantization parameters include a scale factor and a zero point.

[0032] In a possible implementation of the second aspect, the first device processes the neural network model during the debugging phase of the neural network model to obtain the quantization model.

[0033] In a third aspect, the present application also provides a first device, comprising a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute the neural network model quantization method as described in the first aspect.

[0034] In a fourth aspect, the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the neural network model quantization method as described in the first aspect.

[0035] In a fifth aspect, the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the neural network model quantization method described in the first aspect.

[0036] In a sixth aspect, the present application also provides a chip, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the neural network model quantization method described in the first aspect.

[0037] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the neural network model quantization method. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The following is an introduction to the drawings used in the embodiments of the present application.

[0039] Figure 1A It is a schematic diagram of a calculation process of a floating-point convolution calculation unit provided in an embodiment of the present application;

[0040] Figure 1B This is a specific flow chart of dynamic quantification provided by an embodiment of the present application;

[0041] Figure 1Cis a structural schematic diagram of a first device provided in an embodiment of the present application;

[0042] Figure 1D It is a structural diagram of a system architecture provided by an embodiment of the present application;

[0043] Figure 1E is a schematic diagram of an AI framework of a first device provided in an embodiment of the present application;

[0044] Figure 2 It is a flowchart of a neural network model quantization method provided in an embodiment of the present application;

[0045] Figure 3A is a schematic diagram of a quantization process of a first device provided in an embodiment of the present application;

[0046] Figure 3B is a schematic diagram of a calibration mode provided in an embodiment of the present application;

[0047] Figure 4 is a structural schematic diagram of a first device provided in an embodiment of the present application;

[0048] Figure 5 It is a structural schematic diagram of another first device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0050] Due to the applications involved in the embodiments of the present application, in order to facilitate understanding, the relevant terms and other related concepts involved in the embodiments of the present application are first introduced below.

[0051] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0052] The "at least one" mentioned in the embodiments of the present application refers to one or more, and "multiple" refers to two or more. "At least one of the following" or its similar expression refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, c can be single or multiple. "And / or" describes the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can be represented by: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are an "or" relationship. The serial numbers of the steps of the embodiments of the present application (such as step S1, step S21, etc.) are only for distinguishing different steps, and do not limit the order of execution between the steps.

[0053] Furthermore, unless otherwise specified, the ordinal numbers such as "first" and "second" used in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects. For example, the first device and the second device are only for the convenience of description, and do not represent the difference in structure, importance, etc. between the first device and the second device. In some embodiments, the first device and the second device can also be the same device.

[0054] In the above embodiments, the term "when..." can be interpreted as meaning "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modification, equivalent substitution, improvement, etc. made within the concept and principle of the present application shall be included in the protection scope of the present application.

[0055] (1) Neural Network

[0056] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be:

[0057]

[0058] Where, s = 1, 2, ... d, d is a natural number greater than 1, W s For x sThe weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a Sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0059] (2) Deep Neural Networks

[0060] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0061] In summary, the coefficients from the nth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0062] (3) Loss function

[0063] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (Loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this Loss as much as possible.

[0064] (4) Model quantization

[0065] A term in the field of AI model acceleration, referring to compression technology that uses a bit width of less than 32 bits to represent weight data and feature map data in a neural network. The lower the compressed bit width, the more memory space, latency, and power consumption can be saved in actual deployment. Model quantization includes offline quantization and dynamic quantization (Dynamic Quant, DQ). Offline quantization refers to completing the conversion of floating-point models to vector models on the development server before the model is deployed to the end-side device and calculating quantization-related parameters on the server. Depending on the way the quantization parameters are obtained, offline quantization is divided into two types: post-training quantization (PTQ) and quantization-aware training (QAT). The former is characterized by calculating the final quantization parameters by statistically analyzing the data distribution of the neural network model with the calibration set. Only a small amount of unlabeled calibration data sets are needed to complete the quantization process for the pre-trained floating-point model, and it is often used for compression with higher bit widths. The latter is characterized by taking the quantization parameters as part of the model parameters and retraining the neural network model parameters. The latter requires a complete data set for training, and simulates quantization operations during the training process so that the quantization model can further converge to the optimal point. It is often used in the accuracy recovery process after a large loss in the quantization model.

[0066] Specifically, offline quantization means that after the neural network model developer completes the development of the floating-point model, he uses the corresponding quantization toolkit to complete the conversion of the floating-point model to the quantized model in his development environment. During the conversion process, the developer needs to use the test data of the neural network model as the calibration set, calculate the quantization parameters based on the test data distribution, and then generate the corresponding quantized model after completing the parameter calculation, and finally deploy the quantized model in the deployment (production) environment.

[0067] Offline quantization has the following two disadvantages:

[0068] 1. With offline quantization, the inference calculation of the neural network model needs to rely on the third-party framework of machine learning (such as tensorflow, pytroch, etc.). During the quantization process, the quantization parameters of the statistical model must rely on the third-party framework, which means that developers need to rely on the quantization toolkit, need to be aware of the entire quantization process, and need to actively complete the quantization. It is impossible to perform calculations on the hardware where the model is finally deployed. Due to the characteristics of floating-point numbers themselves, different hardware platforms will produce slight differences in the final calculation results.

[0069] 2. The accuracy of the model after quantization is usually obtained through simulation on a quantization server. The accurate accuracy after quantization cannot be obtained immediately after quantization, and the accuracy cannot reflect the final accuracy of the neural network model in the production environment.

[0070] Specifically, dynamic quantization means that the neural network model counts the quantization parameters at the same time during the terminal operation stage. The neural network model calculates the quantization parameters while performing quantization calculations in the deployment (production) environment; specifically, it means that before each fixed-point calculation, the quantization parameters are counted based on the current calculation input. Since dynamic quantization calculates the quantization parameters during the deployment stage of the neural network model, when the calculated quantization parameters are too large, the quantization calculation efficiency will be worse than the original floating-point calculation. For this reason, most terminal models will choose offline quantization.

[0071] The following uses the convolutional computing unit in the neural network model as an example to illustrate dynamic quantization. Figure 1A , Figure 1A It is a schematic diagram of a calculation process of a floating-point convolution calculation unit provided in an embodiment of the present application; Figure 1A Represents the actual calculation process of a floating-point convolution calculation unit. Figure 1B , Figure 1B This is a specific flow chart of dynamic quantization provided by the embodiment of the present application; in the calculation process of dynamic quantization, the quantization parameter calculation unit will first calculate the quantization parameter of the input data, and the calculated quantization parameter and the input data will be simultaneously input into the quantization convolution calculation unit. It can be seen that the benefit of dynamic quantization is that the overall performance overhead of the quantization parameter calculation unit plus the quantization convolution calculation unit is lower than that of the floating-point convolution calculation unit.

[0072] Dynamic quantization has the following two disadvantages:

[0073] 1. When the processing data dimension is too large, the performance overhead of the quantization parameter calculation unit plus the quantization convolution calculation unit will be greater than the floating-point convolution calculation unit, resulting in deterioration of performance after quantization.

[0074] 2. Dynamic quantization can only calculate quantization parameters based on the current frame data, which may lead to deterioration of accuracy and the risk of uncontrollable accuracy.

[0075] Therefore, an embodiment of the present application provides a neural network model quantization method, which can be automatically executed by a first device that deploys the neural network model, and the quantization process does not rely on a quantization toolkit.

[0076] The neural network model quantization method of the embodiment of the present application can be executed by the first device, or by a chip in the first device. The first device is a device for deploying a neural network model.

[0077] The first device may also be referred to as a terminal device, user equipment (UE), a mobile station, a mobile terminal, etc. The first device can be widely used in various scenarios, for example, device-to-device (D2D), vehicle to everything (V2X) communication, machine-type communication (MTC), Internet of Things (IOT), virtual reality, augmented reality, industrial control, automatic driving, telemedicine, smart grid, smart furniture, smart office, smart wear, smart transportation, smart city, etc. The first device may be a mobile phone, a tablet computer, a computer with wireless transceiver function, a wearable device, a vehicle, a drone, a helicopter, an airplane, a ship, a robot, a mechanical arm, a smart home device, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the first device.

[0078] The following first introduces an exemplary first device provided in an embodiment of the present application.

[0079] Figure 1C It is a structural diagram of the first device provided in an embodiment of the present application.

[0080] The following is a detailed description of the embodiment using the first device as an example. It should be understood that the first device may have Figure 1C More or fewer components may be shown, two or more components may be combined, or there may be a different configuration of components. Figure 1C The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0081] The first device may include: a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a Subscriber Identification Module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, a multispectral sensor (not shown), etc.

[0082] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0083] The controller may be the nerve center and command center of the first device. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0084] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0085] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and / or a USB interface, etc.

[0086] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL).

[0087] The I2S interface can be used for audio communication.

[0088] The PCM interface can also be used for audio communication to sample, quantize and encode analog signals.

[0089] The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication.

[0090] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like.

[0091] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or a data signal.

[0092] The SIM interface can be used to communicate with the SIM card interface 195 to implement the function of transmitting data to the SIM card or reading data in the SIM card.

[0093] The USB interface 130 is an interface that complies with USB standard specifications, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.

[0094] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the first device. In other embodiments of the present application, the first device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0095] The charging management module 140 is used to receive charging input from a charger.

[0096] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110 to provide power for the external memory, the display screen 194, the camera 193, and the wireless communication module 160, etc.

[0097] The wireless communication function of the first device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0098] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the first device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization rate of the antennas.

[0099] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the first device. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (Low Noise Amplifier, LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0100] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0101] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), infrared technology (IR), etc. applied on the first device.

[0102] In some embodiments, the antenna 1 of the first device is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the first device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), etc.

[0103] The first device implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0104] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the first device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0105] The first device can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0106] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light signal is transmitted to the camera photosensitive element through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise and brightness of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193. The photosensitive element can also be called an image sensor.

[0107] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the first device may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0108] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the first device is processing a voice signal, the digital signal processor is used to perform Fourier transform on the voice signal.

[0109] The video codec is used to compress or decompress digital video. The first device may support one or more video codecs. Thus, the first device may play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0110] NPU is a neural network computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and continuously self-learn. NPU can realize applications such as intelligent cognition of the first device, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0111] The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the first device.

[0112] The internal memory 121 may be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the first device by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area.

[0113] The first device can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor, etc. For example, music playing, recording, etc. In this embodiment, the first device may include n microphones 170C, where n is a positive integer greater than or equal to 2.

[0114] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals.

[0115] The ambient light sensor 180L is used to sense the brightness of the ambient light. The first device can adaptively adjust the brightness of the display screen 194 according to the sensed brightness of the ambient light. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures.

[0116] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects.

[0117] In an embodiment of the present application, the processor 110 can call computer instructions stored in the internal memory 121 to enable the first device to execute the neural network model quantization method in the embodiment of the present application.

[0118] The neural network model referred to in the embodiments of the present application is a trained neural network model. Therefore, the training process of the neural network model and a system architecture provided by the embodiments of the present application are introduced below.

[0119] See attached Figure 1D , the present application embodiment provides a system architecture. Figure 1D In the system architecture shown, the data acquisition device 100 is used to collect training data for the neural network model. In this embodiment, the training data is determined by the processing task of the neural network model. Different processing tasks correspond to different training data. The above processing tasks include image classification, target detection, video processing, machine translation, sound recognition, etc. For example, when the neural network model is an image classification model, the training data is image data. When the neural network model is a sound recognition model, the training data is sound data. The data acquisition device 100 stores the training data in the database 118.

[0120] The training device 119 can train the initial neural network model based on the training data maintained in the database 118 to obtain the neural network model 114. Exemplarily, in the embodiment of the present application, after the model training is completed, the neural network model 114 can obtain the processing capacity corresponding to the processing task. The training device 119 can be a server or a cloud service device, etc., and can also be a mobile phone terminal, a tablet computer, a laptop computer, AR / VR, a vehicle terminal, a monitoring device, a vehicle-mounted automatic driving system, a drone and other devices.

[0121] It should be noted that, in actual applications, the training data maintained in the database 118 may not all come from the data acquisition device 100, but may also be received from other devices, such as a camera based on an image sensor. It should also be noted that the training device 119 may not necessarily perform model training entirely based on the training data maintained in the database 118, but may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0122] The neural network model 114 obtained after processing by the training device 119 can be applied to the system or device deployed in the terminal, such as Figure 1DThe terminal device 115 shown in the figure can process the input data based on the neural network model 114 to obtain the corresponding processing results. The terminal device 115 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, AR / VR, a vehicle terminal, a vehicle computing platform, a vehicle domain controller, a monitoring device, a vehicle automatic driving system, a drone, etc., and can also be a server or a cloud device. The training device 119 and the terminal device 115 can be the same device. Among them, the terminal device 115 is the first device in the embodiment of the present application.

[0123] In the attached Figure 1D In the figure, the terminal device 115 is configured with an I / O interface 112 for data interaction with external devices. The user can input data to the I / O interface 112 through the client device 117, which can be input by the user or from the database 118; wherein the client device 117 can be an image acquisition device, such as an ordinary camera.

[0124] Optionally, the preprocessing module 113 is used to preprocess the input data received by the I / O interface 112. In the embodiment of the present application, the preprocessing module 113 is used to preprocess the input data received by the I / O interface 112, and the preprocessed data enters the calculation module 111. Exemplarily, the input data may be image data, and the preprocessing module 113 may be used to perform preprocessing such as image enhancement and denoising on the image data.

[0125] When the terminal device 115 preprocesses the input data, or when the computing module 111 of the terminal device 115 performs calculations and other related processing, the terminal device 115 can call the data, code, etc. in the data storage system 116 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 116.

[0126] Finally, the I / O interface 112 returns the processing result corresponding to the input data to the client device 117, thereby providing it to the user. In this case, the client device 117 may be a display.

[0127] In the attached Figure 1D In the case shown in , the user can manually input data, which can be operated through the interface provided by the I / O interface 112. In another case, the client device 117 can automatically send input data to the I / O interface 112. If the client device 117 is required to automatically send input data, it needs to obtain the user's authorization, and the user can set the corresponding permissions in the client device 117. The user can view the results output by the terminal device 115 on the client device 117, and the specific presentation form can be display, sound, action, etc. The client device 117 can also be used as a data collection terminal to collect data such as Figure 1D The input data of the input I / O interface 112 and the output result of the output I / O interface 112 are taken as new sample data and stored in the database 118. Of course, it is also possible not to collect through the client device 117, but to directly use the I / O interface 112 as shown. Figure 1D The input data of the input I / O interface 112 and the output result of the output I / O interface 112 are stored in the database 118 as new sample data.

[0128] It is worth noting that Figure 1D This is only a schematic diagram of a system architecture provided by an embodiment of the present application. Figure 1D The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 1D In the embodiment, the data storage system 116 is an external storage relative to the terminal device 115. In other cases, the data storage system 116 can also be placed in the terminal device 115.

[0129] In a possible implementation, the neural network model quantization method is applied to the debugging stage of the neural network model. When the neural network model is deployed to the terminal side, various debugging needs to be performed first, and the quantization of the neural network model is completed during the debugging period.

[0130] The following is an introduction to the AI ​​framework of the first device of an embodiment of the present application.

[0131] refer to Figure 1E , Figure 1E This is a schematic diagram of an AI framework of a first device provided in an embodiment of the present application; the AI ​​framework of the first device includes a top-level application, an AI engine, an AI foundation unit, a driver layer, a hardware layer, and a corresponding tool chain. The hardware layer includes a central processing unit (CPU), a GPU, a DSP, an NPU, etc.

[0132] Among them, the top-level application is an independent application developed by the algorithm developer and running in the system.

[0133] The AI ​​engine provides developers with the ability to call the end-device system through the Application Programming Interface (API).

[0134] The AI ​​basic unit is responsible for processing the upper-level application instructions transmitted by the AI ​​engine. The main responsibility of the model building module in the AI ​​basic unit is to convert the external three-party model into an internal intermediate representation (IR) diagram of the AI ​​foundation and finally realize the reasoning of the model through the model running module. The reasoning instructions of the model running module will send all computing tasks to the final hardware device through the driver layer. In the entire architecture, the interaction between developers and various models is carried out through the tool chain.

[0135] And in Figure 1E The model quantization module in the AI ​​basic unit in the AI ​​framework shown executes the quantization method of the embodiment of the present application. When the top-level application calls the AI ​​capability, a quantization interface is provided. It only needs to pass the original neural network model into the quantization interface to automatically complete the quantization on the end side and generate the corresponding quantization model. Therefore, in the embodiment of the present application, automatic quantization is implemented on the end side, and an API is provided on the end side system. The developer is unaware of it and it is extremely easy to use.

[0136] The following specifically introduces the neural network model quantization method in the embodiment of the present application. In the following, the execution subject of the neural network model quantization method takes the first device as an example.

[0137] refer to Figure 2 , Figure 2 200 is a flowchart of a neural network model quantization method provided in an embodiment of the present application; the neural network model quantization method 200 includes the following steps:

[0138] 201. The first device obtains a neural network model.

[0139] Specifically, the above neural network model is a neural network model whose data type is floating point number.

[0140] 202. The first device performs inference on the neural network model based on the test set to obtain activation value tensors and / or weight tensors of quantifiable operators in the neural network model.

[0141] Specifically, the test data in the test set is input into the neural network model for inference to obtain the activation value tensor and / or weight tensor of the quantizable operator in the above neural network model. The specific type of the above quantizable operator can be set according to the actual situation of the neural network model quantization method, without special limitation.

[0142] 203. The first device obtains a quantized model of the neural network model based on an activation value tensor and / or a weight tensor of a quantizable operator.

[0143] In an embodiment of the present application, a neural network is inferred based on a test set to obtain an activation value tensor and / or a weight tensor of a quantizable operator, and then the neural network model can be subjected to activation value quantization processing and / or weight quantization processing based on the activation value tensor and / or weight tensor of the quantizable operator to obtain a quantized model corresponding to the neural network model. The quantization method of an embodiment of the present application can be automatically executed by a first device that deploys a neural network model, and the quantization process does not rely on a quantization toolkit, has high ease of use, and has low quantization risk.

[0144] In a possible implementation manner, the above test set includes at least one test data and a label corresponding to each test data; Figure 2 , the neural network model quantization method 200 further includes the following steps:

[0145] 204. The first device processes the test data in the test set by using the quantization model to obtain a processing result.

[0146] Specifically, the test data in the test set is input into the quantization model for inference to obtain the corresponding processing results.

[0147] 205. The first device determines the accuracy of the quantization model based on the processing result and the label corresponding to the test data.

[0148] In an embodiment of the present application, a quantization model is used to process test data to obtain corresponding processing results, and then the accuracy of the quantization model, that is, the accuracy of this quantization, can be determined based on the processing results and the labels corresponding to the test data. The quantization accuracy indicates the degree of closeness between the processing results and the labels of the test data, that is, the performance gap between the quantization model and the original neural network model. For example, the more the processing results of the test data in the test set are the same as the labels corresponding to the test data, the higher the quantization accuracy. The embodiment of the present application completes the quantization calculation in the end-side inference engine, and can obtain the quantization accuracy of the quantization model in the production environment in real time.

[0149] Specifically, when the neural network model is a classification model, the accuracy parameters of the above-mentioned quantization model may be recall (also called recall rate) and / or precision (also called precision rate). Recall rate refers to the proportion of positive samples correctly judged by the classification model to the total positive samples, that is, how many of all positive samples are judged as positive samples by the classification model. And precision rate refers to the proportion of true positive samples among the samples judged as positive by the classification model, that is, how many of all samples judged as positive by the classification model are true positive samples.

[0150] When the neural network model is a segmentation model (such as a semantic segmentation model, an instance segmentation model, or a panoptic segmentation model), the accuracy parameter of the above quantization model may be the mean intersection over union (MIoU), etc. The intersection over union (IoU) is the ratio of the intersection and union of the prediction result and the true value of a certain category by the segmentation model. The mean intersection over union is the average of the intersection over union ratios of all categories.

[0151] In a possible implementation manner, the neural network model quantization method 200 further includes:

[0152] When the accuracy of the quantization model does not meet the accuracy condition, the first device obtains a new test set.

[0153] Specifically, the accuracy condition can be set according to the actual situation. For example, when the accuracy parameter of the quantization model is the recall rate and / or the accuracy rate, when the recall rate and / or the accuracy rate of the quantization model meets certain conditions, it can be determined that the accuracy of the quantization model does not meet the accuracy condition. When the accuracy parameter of the quantization model is the average intersection-over-union ratio, when the average intersection-over-union ratio of the quantization model meets certain conditions, it can be determined that the accuracy of the quantization model does not meet the accuracy condition.

[0154] In the embodiment of the present application, when the accuracy of the quantization model does not meet the accuracy requirement, the first device obtains a new test set and repeats the execution according to the new test set. Figure 2 The steps shown can generate a new quantization model and continue to obtain the accuracy corresponding to the new quantization model until the accuracy of the quantization model meets the accuracy condition, then stop quantization, and the first device uses the final quantization model for inference.

[0155] In a possible implementation manner, the above step 203 specifically includes the following steps:

[0156] 231. The first device obtains key quantization parameters of the neural network model based on the activation value tensor and / or weight tensor of the quantizable operator.

[0157] Specifically, the first device can generate a quantization model based on the activation value tensor and / or weight tensor of the quantizable operator using a quantization algorithm, wherein the quantization key parameter is an intermediate parameter of the quantization algorithm during the quantization process.

[0158] Exemplarily, the key quantization parameters include a scale factor and a zero point. The scale factor and the zero point are used to map the data to be quantized to the distribution of the target quantized data. The data type of the scale factor is a floating point number, which is used to scale the floating point data. The data type of the zero point is an integer number, which is used to ensure that the real zero in the floating point data space is accurately expressed.

[0159] 232. The first device generates a quantization model based on the quantization key parameters and the neural network model.

[0160] Specifically, the quantization algorithm can generate a corresponding quantization model according to the quantization key parameters and the neural network model, and there is no limitation on the specific quantization algorithm.

[0161] In another possible implementation, the neural network model quantization method 200 further includes:

[0162] When the accuracy of the quantization model does not meet the accuracy condition, the first device changes the quantization algorithm and re-executes step 203.

[0163] Specifically, when the accuracy of the quantization model does not meet the accuracy condition, the quantization algorithm in step 203 can be replaced with another quantization algorithm to generate a new quantization model, and then the accuracy corresponding to the new quantization model is obtained until the accuracy of the quantization model meets the accuracy condition. Then, quantization is stopped, and the first device uses the final quantization model for inference.

[0164] The following is a description of the implementation process of the neural network model quantization method of the embodiment of the present application.

[0165] After completing the development of the floating-point neural network model, the AI ​​algorithm developer will integrate the model into the corresponding end-side device and complete the final debugging in the integrated link. The embodiment of the present application will utilize the debugging path of the integrated link to quantize the model during the debugging phase, and the quantization process will be completed when the integrated link debugging is completed. After the quantization is completed, the neural network model quantization method of the embodiment of the present application will generate a quantized file on the end side, and the developer can decide whether to use the quantized model based on the quantized accuracy value.

[0166] refer to Figure 3A , Figure 3AIt is a schematic diagram of the quantization process of a first device provided in an embodiment of the present application; the embodiment of the present application divides quantization into two stages, namely, calibration mode (Calibration Mode, CM) and deployment mode (DeployMode, DM). The calibration mode corresponds to the above-mentioned debugging stage, and the deployment mode corresponds to the stage in which the first device uses the quantization model. During the calibration mode, the developer inputs the trained neural network model and test set, and calls the automatic quantization algorithm of the embodiment of the present application on the end side (this API can be merged with the inference API of the floating-point neural network model and controlled by the incoming variables). After the quantization is completed, the quantization model will be saved on the first device and the quantized accuracy will be output. This allows the developer to automatically complete the quantization process in the first device and obtain the accuracy of the quantized model in real time. The quantization model generated in the calibration mode will be called in the deployment mode to realize the use of the quantization model for end-side model reasoning in the production environment.

[0167] The following is a detailed description of the neural network model quantization method in the calibration mode. Figure 3B , Figure 3B Schematic diagram of a calibration mode provided in an embodiment of the present application; specifically comprising the following steps:

[0168] Step 1. Integrate the neural network model into the end-side device, that is, the first device. The data type of the neural network model is a floating point number.

[0169] Step 2. The first device traverses the neural network model and saves all quantifiable operators to Q ls In the embodiment of the present application, a quantizable operator refers to an operator that can be supported by a terminal device to calculate using a quantized data type.

[0170] Step 3. The first device uses the test set to infer the neural network model and Q ls The activation value tensor and weight tensor of the operator are saved to the activation value tensor set A respectively. ls and the weight tensor set W ls .

[0171] Step 4. Based on A through quantization algorithm ls With W ls Calculate the key parameters of quantization, such as the proportional factor and zero point corresponding to each activation value tensor, and the proportional factor and zero point corresponding to each weight tensor. The calculation results are the proportional factor set S corresponding to the activation value tensor set a , the scale factor set S corresponding to the weight tensor set w , the zero point set O corresponding to the activation value tensor set a The zero point set O corresponding to the weight tensor set w .

[0172] Step 5. S generated by quantization algorithm, neural network model and Step 4 a O a With S w O w Generate a quantized model. The data type of the quantized model is fixed-point.

[0173] Step 6. Input the test data of the test set into the generated quantization model to obtain the corresponding processing results. The accuracy of the quantization model can be obtained based on the processing results and the labels of the test data.

[0174] Step 7. End.

[0175] For example, the calculation process of step 4 in the above steps can be described as the following formula:

[0176]

[0177]

[0178] In the above formula, asbmax() is used to find the maximum absolute value of the input data, max() is used to find the maximum value of the input data, min() is used to find the minimum value of the input data, and bits refers to the target number of bits for quantization. For example, when quantization is performed in the INT8 quantization model, bits = 8.

[0179] It should be noted that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced mutually, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0180] The method of the embodiment of the present application is described in detail above, and the device provided by the embodiment of the present application is introduced below.

[0181] Figure 4 A schematic diagram of a possible device structure provided in an embodiment of the present application. Figure 4 The first device shown can be used to implement the functions of the above-mentioned neural network model quantization method embodiment, and thus can also achieve the beneficial effects of the above-mentioned neural network model quantization method embodiment. In an embodiment of the present application, the first device can be an electronic device, or a module (such as a chip) applied to an electronic device. The first device is a device for deploying a neural network model.

[0182] like Figure 4 As shown, the first device 400 includes an acquisition module 401 and an inference module 402. The first device 400 is used to implement the above Figure 2Alternatively, the first device 400 may include a device for implementing the above Figure 2 A module for any function or operation in the neural network model quantization method embodiment shown in the figure may be implemented in whole or in part by software, hardware, firmware or any combination thereof.

[0183] When the first device 400 is used to implement Figure 2 The functions in the method embodiment shown are as follows: an acquisition module 401 is used to acquire a neural network model. An inference module 402 is used to infer the neural network model based on a test set to acquire an activation value tensor and / or a weight tensor of a quantizable operator in the neural network model. The acquisition module 401 is also used to acquire a quantized model of the neural network model based on the activation value tensor and / or the weight tensor of the quantizable operator.

[0184] The above neural network model is a floating-point neural network model.

[0185] In an embodiment of the present application, the first device infers the neural network based on the test set to obtain the activation value tensor and / or weight tensor of the quantizable operator, and performs activation value quantization processing and / or weight quantization processing on the neural network model based on the activation value tensor and / or weight tensor of the quantizable operator to obtain a quantization model corresponding to the neural network model. In an embodiment of the present application, the quantization method of the present solution can be executed by the first device that deploys the neural network model to achieve automatic quantization, and the quantization process does not rely on the quantization toolkit, and is easy to use.

[0186] In a possible implementation manner, the test set includes test data and labels corresponding to the test data; the first device 400 further includes a processing module 403 and a determination module 404, wherein:

[0187] The processing module 403 is used to process the test data in the test set through the quantization model to obtain a processing result.

[0188] The determination module 404 is used to determine the accuracy of the quantization model based on the processing results and the labels corresponding to the test data.

[0189] In a possible implementation, the acquisition module 401 is further configured to acquire a new test set when the accuracy of the quantization model does not meet the accuracy requirement. When the accuracy of the quantization model does not meet the accuracy requirement, a new test set is acquired to generate a new quantization model.

[0190] In a possible implementation manner, the acquisition module 401 is specifically used to:

[0191] The key quantization parameters of the neural network model are obtained based on the activation value tensor and / or weight tensor of the quantizable operator.

[0192] Generate a quantization model based on quantization key parameters and neural network model.

[0193] In a possible implementation manner, the above-mentioned key quantization parameters include a scale factor and a zero point.

[0194] In a possible implementation, the first device processes the neural network model during the debugging phase of the neural network model to obtain the quantization model.

[0195] For the introduction of the above modules, please refer to the description of the above embodiments, which will not be repeated here.

[0196] refer to Figure 5 , Figure 5 500 includes a memory 501, a processor 502, a communication interface 504, and a bus 503. The memory 501, the processor 502, and the communication interface 504 are connected to each other through the bus 503.

[0197] Optionally, the first device 500 further includes a display (not shown), which is connected to the memory 501, the processor 502, and the communication interface 504 via the bus 503. The display is used to output images. Optionally, the first device further includes an output module (not shown), which is connected to the memory 501, the processor 502, and the communication interface 504 via the bus 503, and the output module is used to output audio. The output module can be a speaker. Exemplarily, after the first device 500 determines the final quantization model, the quantization end prompt information can be output through the output module, such as voice output or display output.

[0198] The memory 501 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 501 may store a program. When the program stored in the memory 501 is executed by the processor 502, the processor 502 and the communication interface 504 are used to execute the various steps of the neural network model quantization method of any embodiment of the present application.

[0199] The processor 502 can adopt a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the first device of any embodiment of the present application, or to execute the neural network model quantization method of any embodiment of the present application.

[0200] The processor 502 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the neural network model quantization method of any embodiment of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 502. The above-mentioned processor 502 may also be a general-purpose processor, a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the neural network model quantization method in combination with any embodiment of the present application may be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 501, and the processor 502 reads the information in the memory 501, and combines its hardware to complete the functions required to be performed by the unit included in the first device of any embodiment of the present application, or executes the neural network model quantization method of any embodiment of the present application.

[0201] The communication interface 504 uses a transceiver such as, but not limited to, a transceiver to implement communication between the first device 500 and other devices or a communication network. For example, a neural network model can be obtained through the communication interface 504.

[0202] The bus 503 may include a path for transmitting information between various components of the first device 500 (eg, the memory 501 , the processor 502 , and the communication interface 504 ).

[0203] It should be noted that although Figure 5The first device 500 shown only shows a memory 501, a processor 502, a bus 503, and a communication interface 504. However, in the specific implementation process, those skilled in the art should understand that the first device 500 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the first device 500 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the first device 500 may also only include the devices necessary for implementing the embodiments of the present application, and does not necessarily include Figure 5 All devices shown in .

[0204] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0205] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0207] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0208] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A neural network model quantization method, characterized in that: The method is applied to a first device, where the first device is a device for deploying the neural network model, and the method comprises the following steps: Obtaining the neural network model; Performing reasoning on the neural network model based on a test set to obtain activation value tensors and / or weight tensors of quantizable operators in the neural network model; A quantization model of the neural network model is obtained based on the activation value tensor and / or weight tensor of the quantizable operator.

2. The method according to claim 1, characterized in that The test set includes test data and labels corresponding to the test data; the method further includes: Processing the test data in the test set by the quantitative model to obtain a processing result; The accuracy of the quantization model is determined based on the processing results and the labels corresponding to the test data.

3. The method according to claim 2, characterized in that The method further comprises: When the accuracy of the quantization model does not meet the accuracy condition, a new test set is obtained.

4. The method according to any one of claims 1 to 3, characterized in that: The step of obtaining the quantization model of the neural network model based on the activation value tensor and / or weight tensor of the quantizable operator includes: Acquire quantization key parameters of the neural network model based on the activation value tensor and / or weight tensor of the quantizable operator; The quantization model is generated based on the quantization key parameters and the neural network model.

5. The method according to claim 4, characterized in that The key quantization parameters include a scale factor and a zero point.

6. A first device, characterized in that: The first device is a device for deploying the neural network model, and the first device includes: An acquisition module, used to acquire the neural network model; An inference module, configured to perform inference on the neural network model based on a test set to obtain an activation value tensor and / or a weight tensor of a quantizable operator in the neural network model; The acquisition module is also used to acquire the quantization model of the neural network model based on the activation value tensor and / or weight tensor of the quantizable operator.

7. The device according to claim 6, characterized in that The test set includes test data and labels corresponding to the test data; the first device also includes: A processing module, used for processing the test data in the test set by using the quantization model to obtain a processing result; A determination module is used to determine the accuracy of the quantization model based on the processing results and the labels corresponding to the test data.

8. The device according to claim 7, characterized in that The acquisition module is also used to acquire a new test set when the accuracy of the quantization model does not meet the accuracy condition.

9. The device according to any one of claims 6 to 8, characterized in that In terms of acquiring the quantized model of the neural network model based on the activation value tensor and / or weight tensor of the quantizable operator, the acquisition module is specifically used to: Acquire quantization key parameters of the neural network model based on the activation value tensor and / or weight tensor of the quantizable operator; The quantization model is generated based on the quantization key parameters and the neural network model.

10. The device according to claim 9, characterized in that The key quantization parameters include a scale factor and a zero point.

11. A first device, characterized in that: It includes a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute the neural network model quantization method as described in any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the neural network model quantization method as described in any one of claims 1 to 5.

Images