A model quantization method and apparatus thereof

Abstract

The application discloses a model quantization method, and relates to the field of artificial intelligence, which comprises the following steps: acquiring a first feature map of a first intermediate layer output of a neural network; determining a first truncation interval satisfying a preset condition according to the numerical distribution of a plurality of first feature points in the first feature map; the first truncation interval comprises a first upper boundary threshold and a first lower boundary threshold, and the preset condition comprises that the numerical distribution density of the feature points in the numerical truncation interval is greater than the numerical distribution density of the feature points outside the numerical truncation interval. The application uses an upper end truncation threshold and a lower end truncation threshold to represent the parameter setting of quantization, instead of the zero position and the range size commonly used in previous schemes, uses double-end cutting based on density on the floating-point model first, removes outliers in the long-tail distribution, can adapt to the asymmetric distribution trend, and further improves the precision of the quantized model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a model quantization method and device thereof. BACKGROUND

[0002] Neural network models, especially those implementing vision-related tasks, require a large amount of memory space and computing resources when actually running, which makes it very difficult to deploy them on mobile devices.

[0003] In order to improve running efficiency, various methods are used to compress the size of the model, such as network pruning, model quantization, lightweight architecture design, and knowledge distillation. Among these methods, model quantization is a relatively better technique for existing artificial intelligence acceleration chips because these chips usually focus on low-precision calculations, thereby significantly reducing the latency, memory usage, and power consumption of model inference. However, the accuracy of the compressed model obtained by the model quantization method in the prior art is reduced. SUMMARY

[0004] The present application provides a model quantization method and related device, which can improve the accuracy of the network.

[0005] In a first aspect, an embodiment of the present application provides a model quantization method, which comprises: obtaining a first feature map, the first feature map being a feature map output by a first intermediate layer of a neural network, the first feature map comprising a plurality of first feature points, and the neural network being a floating-point model; determining a first truncation interval satisfying a preset condition according to a numerical distribution of the plurality of first feature points; the first truncation interval comprising a first upper boundary threshold and a first lower boundary threshold, wherein when quantizing the feature map output by the first intermediate layer, the numerical value of a first feature point less than the first lower boundary threshold is quantized to the first lower boundary threshold, and the numerical value of a first feature point greater than the first upper boundary threshold is quantized to the first upper boundary threshold; and the preset condition comprising that the numerical distribution density of the feature points within the numerical truncation interval is greater than the numerical distribution density of the feature points outside the numerical truncation interval. The present application uses an upper end truncation threshold and a lower end truncation threshold to represent the parameter settings of quantization, rather than the zero position and range size commonly used in previous schemes. The floating-point model is first truncated at both ends based on the density, removing outliers in the long-tail distribution, which can adapt to the asymmetric distribution trend and thus improve the accuracy of the quantized model.

[0006] In a possible implementation, the preset condition further comprises that the number of feature points within the numerical truncation interval and the proportion of the number of feature points included in the feature map are greater than a first threshold.

[0007] In a possible implementation, the determining, according to the numerical distribution of the plurality of first feature points, of the first truncation interval that satisfies the preset condition includes: dividing a numerical range of the plurality of first feature points into a plurality of numerical intervals according to numerical values; and determining, from both sides of the plurality of numerical intervals to the inside in turn, a numerical interval with a smaller numerical distribution density as an edge numerical interval until a quantity of first feature points in a numerical interval other than the edge numerical interval in the plurality of numerical intervals and a quantity ratio of the plurality of first feature points are less than a second threshold value, and determining the numerical interval other than the edge numerical interval as the first truncation interval.

[0008] The distribution of the feature map output by the intermediate layer of the model is usually shown as dense in the middle and sparse at both ends, so the dense region is far away from the original boundary, which is very unfriendly to model quantization, especially for low bits. Therefore, the embodiment of the present application proposes to cut off the outliers of the feature map based on the density of the double-end truncation to help reduce the distribution to the effective range. By continuously comparing the distribution density of the upper and lower ends, the outliers in the floating-point parameters are removed on the basis of considering the long tail and asymmetric distribution of the feature map, and a floating-point model with relatively rough upper and lower boundaries is obtained.

[0009] In a possible implementation, the first feature map is a feature map output by a first intermediate layer of a neural network when processing a first batch of training samples; and the method further includes: obtaining a second feature map, the second feature map being a feature map output by the first intermediate layer of the neural network when processing a second batch of training samples, the second feature map including a plurality of second feature points; determining, according to a numerical distribution of the plurality of second feature points, a second truncation interval that satisfies the preset condition; the second truncation interval including a second upper boundary threshold and a second lower boundary threshold; and updating, according to the second truncation interval, the first truncation interval to obtain a third truncation interval. Different truncation intervals can be determined based on the numerical distribution of the output feature map of different intermediate layers in the neural network.

[0010] In a possible implementation, the updating, according to the second truncation interval, of the first truncation interval includes updating, according to the second truncation interval, the first truncation interval by exponential moving average.

[0011] Pixel-aware calibration can be performed, the floating-point parameters obtained in the first step are used to quantize the model, then the full-precision model is used to supervise the low-bit quantization model, and the quantization parameters are further fine-tuned on a small calibration data set, so that the quantization parameters further adapt to the highly dynamic feature map distribution changes in the fine-tuning process.

[0012] In a possible implementation, the method further includes: obtaining a third feature map, the third feature map being a feature map output by a second intermediate layer of a neural network, the third feature map including a plurality of third feature points, and the neural network being a floating-point model; determining, according to a numerical distribution of the plurality of third feature points, a third clipping interval that satisfies the preset condition; the third clipping interval including a third upper boundary threshold and a third lower boundary threshold, wherein, when quantizing the feature map output by the second intermediate layer, a numerical value of a third feature point less than the third lower boundary threshold is quantized to the third lower boundary threshold, and a numerical value of a third feature point greater than the third upper boundary threshold is quantized to the third upper boundary threshold.

[0013] In a possible implementation, the first clipping interval is used to quantize the neural network to obtain a quantized neural network; the method further includes: obtaining a first output and a second output, the first output being an output of an intermediate layer or an output layer of the neural network when processing a training sample, and the second output being an output of an intermediate layer or an output layer of the quantized neural network when processing the training sample; determining a loss according to the first output and the second output, and updating the first clipping interval according to the loss. The strategy of using pixel perception calibration for the model is used to appropriately fine-tune the quantization parameter, so that the quantized model can better adapt to high dynamic feature changes.

[0014] In a possible implementation, the training sample is unlabeled data.

[0015] The present application only needs a small amount of unlabeled calibration data, does not need training, and can obtain a quantized model in a few minutes. Compared with quantization perception training, the present application embodiment does not need a complete training data set and high-cost training server resources, greatly reduces the cost required by the model quantization process, and can obtain the structure and parameters of the quantized model in a short time, greatly speeding up the efficiency of model online deployment.

[0016] In a possible implementation, the first output and the second output are feature maps output by intermediate layers; the determining a loss according to the first output and the second output includes: calculating L2 norms of the first output and the second output respectively to obtain processed first output and processed second output; and determining a loss according to a mean square error between the processed first output and the processed second output.

[0017] In a possible implementation, the updating the first clipping interval according to the loss includes:

[0018] The first clipping interval is updated according to the loss while keeping the clipping interval corresponding to the weight parameter of the neural network unchanged.

[0019] In a second aspect, the present application provides a model quantization device, the device comprising:

[0020] an acquisition module configured to acquire a first feature map, the first feature map being a feature map output by a first intermediate layer of a neural network, the first feature map comprising a plurality of first feature points, the neural network being a floating-point model;

[0021] a processing module configured to determine, according to a numerical distribution of the plurality of first feature points, a first numerical range satisfying a preset condition, the first numerical range comprising a first upper boundary threshold and a first lower boundary threshold, wherein, when quantizing the feature map output by the first intermediate layer, a numerical value of a first feature point less than the first lower boundary threshold is quantized to the first lower boundary threshold, and a numerical value of a first feature point greater than the first upper boundary threshold is quantized to the first upper boundary threshold, the preset condition comprising that a numerical distribution density of a feature point within the numerical range is greater than a numerical distribution density of a feature point outside the numerical range.

[0022] In a possible implementation, the preset condition further comprises:

[0023] a number of feature points within the numerical range and a proportion of feature points included in the feature map are greater than a first threshold.

[0024] In a possible implementation, the processing module is specifically configured to:

[0025] divide a numerical range of the plurality of first feature points into a plurality of numerical ranges according to numerical values;

[0026] determine, from both sides of the plurality of numerical ranges to the inside in turn, a numerical range with a smaller numerical distribution density as an edge numerical range, until a number of first feature points in a numerical range other than the edge numerical range among the plurality of numerical ranges and a proportion of the plurality of first feature points are less than a second threshold, and determine the numerical range other than the edge numerical range as the first numerical range.

[0027] In a possible implementation, the first feature map is a feature map output by the first intermediate layer when the neural network processes a first batch of training samples;

[0028] the acquisition module is further configured to:

[0029] acquire a second feature map, the second feature map being a feature map output by the first intermediate layer when the neural network processes a second batch of training samples, the second feature map comprising a plurality of second feature points;

[0030] the processing module is further configured to:

[0031] determine a second truncated interval satisfying the preset condition according to the numerical distribution of the plurality of second feature points; the second truncated interval comprises a second upper boundary threshold and a second lower boundary threshold;

[0032] update the first truncated interval according to the second truncated interval to obtain a third truncated interval.

[0033] In a possible implementation, the processing module is specifically configured to:

[0034] update the first truncated interval by exponential moving average according to the second truncated interval.

[0035] In a possible implementation, the obtaining module is further configured to:

[0036] obtain a third feature map, the third feature map being a feature map output by a second intermediate layer of a neural network, the third feature map comprising a plurality of third feature points, and the neural network being a floating-point model;

[0037] The processing module is further configured to:

[0038] determine a third truncated interval satisfying the preset condition according to the numerical distribution of the plurality of third feature points; the third truncated interval comprises a third upper boundary threshold and a third lower boundary threshold, wherein, when quantizing the feature map output by the second intermediate layer, a numerical value of a third feature point less than the third lower boundary threshold is quantized to the third lower boundary threshold, and a numerical value of a third feature point greater than the third upper boundary threshold is quantized to the third upper boundary threshold.

[0039] In a possible implementation, the first truncated interval is used to quantize the neural network to obtain a quantized neural network; and the obtaining module is further configured to:

[0040] The processing module is further configured to:

[0041] obtain a first output and a second output, the first output being an output of an intermediate layer or an output layer of the neural network when processing a training sample, and the second output being an output of an intermediate layer or an output layer of the quantized neural network when processing the training sample;

[0042] determine a loss according to the first output and the second output, and update the first truncated interval according to the loss.

[0043] In a possible implementation, the training sample is unlabeled data.

[0044] In a possible implementation, the processing module is specifically configured to:

[0045] compute L2-norms of the first output and the second output respectively to obtain a processed first output and a processed second output;

[0046] determine a loss according to a mean square error between the processed first output and the processed second output.

[0047] In a possible implementation, the processing module is specifically configured to:

[0048] update the first truncated interval according to the loss while keeping the truncated interval corresponding to the weight parameter of the neural network unchanged.

[0049] In a third aspect, an embodiment of the present application provides a data processing apparatus, which can include a memory, a processor and a bus system, wherein the memory is configured to store a program, and the processor is configured to execute the program in the memory to perform the method in the first aspect and any optional method thereof.

[0050] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is executed on a computer, the computer is caused to perform the method in the first aspect and any optional method thereof.

[0051] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, causes the computer to perform the method in the first aspect and any optional method thereof.

[0052] In a sixth aspect, the present application provides a chip system, which includes a processor configured to support an execution device or a training device to implement the functions involved in the above aspects, for example, to send or process the data involved in the above method; or, information. In a possible design, the chip system further includes a memory, and the memory is configured to save necessary program instructions and data for the execution device or the training device. The chip system can be composed of a chip, or can include a chip and other discrete devices. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 A structural schematic diagram of an artificial intelligence subject framework;

[0054] Figure 2 A schematic diagram of an application system framework of the present application;

[0055] Figure 3a A kind of application scenario of the present application;

[0056] Figure 3b A kind of application scenario of the present application;

[0057] Figure 4 An application scenario of the present application is shown in the figure;

[0058] Figure 5 An application scenario of the present application is shown in the figure;

[0059] Figure 6 A chip architecture of the present application is shown in the figure;

[0060] Figure 7 A flowchart of a model quantization method provided by an embodiment of the present application is shown in the figure;

[0061] Figure 8A A numerical distribution is shown in the figure;

[0062] Figure 8B A flowchart of a model quantization method provided by an embodiment of the present application is shown in the figure;

[0063] Figure 9 A flowchart of a model quantization method provided by an embodiment of the present application is shown in the figure;

[0064] Figure 10 A flowchart of a model quantization method provided by an embodiment of the present application is shown in the figure;

[0065] Figure 11 An effect diagram of a model quantization method provided by an embodiment of the present application is shown in the figure;

[0066] Figure 12 A structural diagram of a model quantization device provided by an embodiment of the present application is shown in the figure;

[0067] Figure 13 A structural diagram of an execution device provided by an embodiment of the present application is shown in the figure;

[0068] Figure 14 A structural diagram of a training device provided by an embodiment of the present application is shown in the figure;

[0069] Figure 15 A structural diagram of a chip provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0070] The embodiments of the present application are described below in conjunction with the accompanying drawings. The terms used in the embodiment part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0071] The embodiments of the present application are described below in conjunction with the accompanying drawings. It is known to those of ordinary skill in the art that, as technology develops and new scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0072] The terms "first", "second", "third", etc. in the specification and claims of this application and in the above figures are used to distinguish similar objects, and are not necessarily used to describe a particular sequential or chronological order. It should be understood that the terms so used are interchangeable under appropriate circumstances and are merely employed in the description of embodiments of this application for the purpose of differentiation among like objects. Furthermore, the terms "comprising", "having", "including", and any variations thereof in the specification and claims are intended to cover a non-exclusive inclusion, such that a process, method, system, product or apparatus that comprises, has, includes or contains a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, system, product or apparatus.

[0073] First, the overall workflow of the artificial intelligence system is described, please see Figure 1 , Figure 1 A structural diagram of an artificial intelligence subject framework is shown, and the above artificial intelligence subject framework is described below from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0074] (1) Infrastructure

[0075] The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. Hardware acceleration chips); the underlying platform includes distributed computing framework and network related platform guarantee and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication acquire data, which are provided to intelligent chips in the distributed computing system provided by the underlying platform for calculation.

[0076] (2) Data

[0077] The data on the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. Data involves graphics, images, voice, text, and also involves Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, humidity, etc.

[0078] (3) Data processing

[0079] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision, and the like.

[0080] Among them, machine learning and deep learning can model, extract, preprocess, train, and the like of symbolic and formalized intelligent information on data.

[0081] Reasoning refers to the process of simulating human intelligent reasoning methods in a computer or intelligent system, using formalized information to solve problems according to reasoning control strategies, and the typical function is search and matching.

[0082] Decision refers to the process of decision-making after intelligent information is reasoned, and generally provides functions such as classification, sorting, and prediction.

[0083] (4) General capabilities

[0084] After data is processed as mentioned above, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, and the like.

[0085] (5) Intelligent products and industry applications

[0086] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields, which are encapsulations of overall solutions of artificial intelligence, and realize application landing by productizing intelligent information decision-making. The application fields mainly include intelligent terminals, intelligent transportation, intelligent medical treatment, automatic driving, smart cities, and the like.

[0087] First, the application scenario of the present application is introduced. The present application can be, but is not limited to, cloud services (model quantization and the like compression services) provided by a cloud-side server.

[0088] In one possible implementation, the server can provide a model quantization and the like neural network compression service for the end side through an application programming interface (application programming interface, API).

[0089] Among them, the terminal device can send relevant parameters (such as compression requirements, etc.) to the server through the API provided by the cloud, and the server can obtain a processing result based on the received parameters, and return the processing result (such as a compressed neural network model) to the terminal.

[0090] In addition, the processing flow of model compression can also be executed on the end side, which is not limited here.

[0091] It should be understood that the model quantization method in the embodiments of the present application involves AI-related operations. The following will be described in combination with an example of a neural network compression method. Figure 2The system architecture provided by the embodiments of the present application is described in detail.

[0092] Figure 2 The system architecture provided by the embodiments of the present application is described in detail. Figure 2 As shown in the figure, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data collection system 560.

[0093] The execution device 510 includes a computing module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The target model / rule 501 can be included in the computing module 511, and the preprocessing module 513 and the preprocessing module 514 are optional.

[0094] The training device 520 can be a device for performing model quantization.

[0095] The data collection device 560 is used to collect training samples. The training samples can be image, audio / video, text, etc. After collecting the training samples, the data collection device 560 stores the training samples in the database 530.

[0096] The training device 520 can obtain the target model / rule 501 by compressing the neural network (such as the neural network in the embodiments of the present application, etc.) to be compressed based on the training samples maintained in the database 530.

[0097] It should be noted that in actual application, the training samples maintained in the database 530 can not all come from the collection of the data collection device 560, but can also be received from other devices. In addition, it should be noted that the training device 520 can not completely compress the target model / rule 501 based on the training samples maintained in the database 530, but can also obtain the training samples from the cloud or other places to perform model compression. The above description should not be regarded as a limitation of the embodiments of the present application.

[0098] The target model / rule 501 obtained by the training device 520 can be applied to different systems or devices, such as the execution device 510 as shown in the figure. The execution device 510 can be a terminal such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle-mounted terminal, etc., and can also be a server, etc. Figure 2

[0099] Specifically, the training device 520 can deliver the compressed model to the execution device 510.

[0100] In the embodiments of the present application, the training device 520 can be a server, and the execution device 510 can be a terminal such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle-mounted terminal, etc. Figure 2 ​In the execution device 510, an input / output (I / O) interface 512 is configured for data interaction with external devices. Users can input data to the I / O interface 512 through the client device 540.

[0101] Preprocessing modules 513 and 514 are used to preprocess the input data received from the I / O interface 512. It should be understood that preprocessing modules 513 and 514 may be absent, or only one preprocessing module may be used. When preprocessing modules 513 and 514 are absent, the calculation module 511 can be used directly to process the input data.

[0102] During the preprocessing of input data by the execution device 510, or during the calculation module 511 of the execution device 510 performing calculations and other related processes, the execution device 510 can call data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0103] Finally, the I / O interface 512 provides the processing result (e.g., the processing result obtained after processing compressed model data based on the method provided in the embodiments of this application) to the client device 540, thereby providing it to the user.

[0104] exist Figure 2 In the illustrated scenario, the user can manually provide input data, which can be done through the interface provided by I / O interface 512. Alternatively, the client device 540 can automatically send input data to I / O interface 512. If user authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permissions in the client device 540. The user can view the output results of the execution device 510 on the client device 540, which can be presented in various forms such as display, sound, or animation. The client device 540 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530. Alternatively, data can be collected directly from the I / O interface 512 without going through the client device 540, using the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530.

[0105] It is worth noting that, Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 2In some embodiments, the data storage system 550 is an external memory to the execution device 510, and in other embodiments, the data storage system 550 can also be placed in the execution device 510. It should be understood that the execution device 510 described above can be deployed in the client device 540.

[0106] From the inference side of the model:

[0107] In the embodiments of the present application, the computing module 511 of the execution device 520 can obtain the code stored in the data storage system 550 to perform the inference of the model.

[0108] In the embodiments of the present application, the computing module 511 of the execution device 520 can include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 can be a hardware system with an execution instruction function, such as a CPU, a DSP, etc., or a hardware system without an execution instruction function, such as an ASIC, an FPGA, etc., or a combination of the hardware system without an execution instruction function and the hardware system with an execution instruction function.

[0109] Specifically, the computing module 511 of the execution device 520 can be a hardware system with an execution instruction function, and the connection relationship prediction method provided in the embodiments of the present application can be a software code stored in the memory. The computing module 511 of the execution device 520 can obtain the software code from the memory and execute the obtained software code to perform the inference of the model.

[0110] It should be understood that the computing module 511 of the execution device 520 can be a combination of a hardware system without an execution instruction function and a hardware system with an execution instruction function, and part of the steps of model inference can also be implemented by the hardware system without an execution instruction function in the computing module 511 of the execution device 520, which is not limited here.

[0111] From the training side of the model:

[0112] In the embodiments of the present application, the training device 520 can obtain the code stored in the memory (not shown in the figure) to implement the steps related to model compression in the embodiments of the present application. Figure 2

[0113] ​In this embodiment of the application, the training device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0114] It should be understood that the training device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps related to the neutralization model compression provided in the embodiments of this application can also be implemented by the hardware system in the training device 520 without the function of executing instructions, which is not limited here.

[0115] The compressed neural network in this application embodiment can be applied to fields that require neural network implementation, such as visual perception tasks, speech and semantic related natural language synthesis tasks, and audio and video processing tasks.

[0116] For example, the compressed neural network in this application embodiment can be applied to fields such as driver assistance, autonomous driving, and mobile terminals that require the completion of multiple perception tasks. A schematic diagram of the application system framework of this application can be shown below. Figure 3a and Figure 3b As shown, the video is frame-by-frame extracted to obtain a single image. This image is then fed into the neural network provided in this embodiment to obtain information such as the 2D and 3D dimensions of the object of interest, the mask, and key points within the image. These detection results are output to a post-processing module for further processing. For example, they can be fed into a planning and control unit in an autonomous driving system for decision-making, or into a beautification algorithm on a mobile terminal to obtain a beautified image. The following sections provide a brief introduction to two application scenarios: ADAS / ADS visual perception systems and mobile beautification.

[0117] Application Scenario 1: ADAS / ADS Visual Perception System

[0118] like Figure 4As shown, in ADAS and ADS, real-time multi-type 2D target detection is required, including: dynamic obstacles (Pedestrian, Cyclist, Tricycle, Car, Truck, Bus), static obstacles (TrafficCone, TrafficStick, FireHydrant, Motocycle, Bicycle), traffic signs (TrafficSign, GuideSign, Billboard, TrafficLight_Red / TrafficLight_Yellow / TrafficLight_Green / TrafficLight_Black, RoadSign). In addition, in order to accurately obtain the area occupied by the dynamic obstacle in the 3D space, 3D estimation of the dynamic obstacle is also required, and a 3D box is output. In order to fuse the data of the laser radar, the mask of the dynamic obstacle needs to be obtained, so as to screen out the laser point cloud hitting the dynamic obstacle; in order to accurately detect the parking space, the four key points of the parking space need to be detected; in order to perform composition positioning, the key points of the static target need to be detected. The neural network corresponding to the technical solution provided by the embodiment of the application can complete all or part of the above functions.

[0119] For example, the compressed model obtained by the technical solution provided by the embodiment of the application can be applied to adaptive cruise in assisted driving and advanced assisted driving.

[0120] The adaptive cruise function in ADAS needs to adjust the speed of the ego vehicle according to the position and speed of the vehicle in front of the lane, so as to realize automatic cruise without collision. When there is no other traffic participant (target) in front of the ego vehicle, the ego vehicle advances according to the pre-set speed or road speed limit. For example, when the perception system of the ego vehicle detects that other traffic participants enter the front of the lane, the ego vehicle will automatically reduce the speed of the ego vehicle according to the position and speed of the white vehicle, so as to avoid collision when the front vehicle decelerates.

[0121] For example, the compressed model obtained by the technical solution provided by the embodiment of the application can be applied to target trajectory prediction in automatic assisted driving and monitoring.

[0122] Trajectory prediction obtains the position, orientation, size, etc. of important traffic participants in the environment through camera sensing and target detection algorithm, and obtains the motion speed and direction of each target through accumulation of multiple frame detection results, so as to realize the prediction of the future motion trajectory of the target and serve as the basis for subsequent automatic driving vehicle decision control. For example, the automatic driving vehicle can predict the future motion direction of surrounding vehicles. For example, in a monitoring scene, the future motion of a pedestrian can be predicted by detecting the orientation and position of the pedestrian, so as to identify the possible crowd form in advance.

[0123] In an unmanned vehicle, it is crucial for the operation of the unmanned vehicle to quickly detect and locate the positions of pedestrians and vehicles in pictures taken by a camera. Since the operation of the unmanned vehicle requires fast response speed, the detection has high real-time requirements. The object detection used in pedestrian and vehicle detection is also based on a neural network, which uses the nonlinear enhanced activation function or the deep reparameterization training scheme proposed in the present application. For example, the nonlinear enhanced activation function and the deep reparameterization training scheme proposed in the present application are applied to the neural network (such as a shallow neural network) to improve the accuracy of the neural network, replace the original deep network, and achieve faster detection speed.

[0124] Application scenario 2: mobile phone beautification function

[0125] As shown in Figure 5 , in a mobile phone, the compressed neural network obtained by the method provided in the embodiments of the present application can detect the mask and key points of a human body, and can enlarge or reduce the corresponding parts of the human body, such as waist-in and hip-up operations, so as to output a beautified picture.

[0126] Application scenario 3: image classification scenario

[0127] After the object recognition device obtains a to-be-classified image, the compressed neural network obtained by the method provided in the present application can obtain the category of the object in the to-be-classified image, and then can classify the to-be-classified image according to the category of the object in the to-be-classified image. For a photographer, many photos will be taken every day, including animal photos, human photos, and plant photos. The method of the present application can quickly classify the photos according to the content in the photos, and can classify the photos into animal-containing photos, human-containing photos, and plant-containing photos.

[0128] Application scenario 4: commodity classification

[0129] After obtaining the image of the commodity, the method provided in the application is used to obtain the category of the commodity in the compressed neural network of the image of the commodity, and then the commodity is classified according to the category of the commodity. For a large number of commodities in a large shopping mall or supermarket, the model quantification method of the application can complete the classification of the commodities, reducing the time cost and labor cost.

[0130] Application scenario 5: natural language processing (NLP):

[0131] Natural language (natural language) is human language, and natural language processing (NLP) is the processing of human language. Natural language processing is a systematic analysis, understanding and information extraction process of text data in an intelligent and efficient way. By using NLP and its components, we can manage very large blocks of text data, or perform a large number of automated tasks, and solve various problems such as automatic summarization, machine translation (MT), named entity recognition (NER), relation extraction (RE), information extraction (IE), sentiment analysis, speech recognition, question answering and topic segmentation.

[0132] For example, natural language processing tasks can be divided into the following categories.

[0133] Sequence labeling: each word in a sentence requires the model to give a classification category according to the context. For example, Chinese word segmentation, part-of-speech tagging, named entity recognition, semantic role labeling.

[0134] Classification task: the entire sentence outputs a classification value, such as text classification.

[0135] Sentence relationship inference: given two sentences, determine whether the two sentences have a certain nominal relationship. For example, entilment, QA, semantic rewriting, natural language inference.

[0136] Generative task: output a piece of text, generate another piece of text. For example, machine translation, text summarization, writing poetry and making sentences, and speaking pictures.

[0137] The following are some examples of natural language processing cases.

[0138] Word segmentation or word breaker (WB): to cut continuous natural language text into a sequence of words with semantic rationality and integrity, which can solve the problem of cross ambiguity.

[0139] Named entity recognition (NER): to identify entities (people, places, organizations, times, works, etc.) with specific meanings in natural language text.

[0140] Part-speech tagging: to assign a part of speech (noun, verb, adjective, etc.) to each word in natural language text; dependency parsing: to automatically analyze the syntactic components (subject, predicate, object, attribute, adverbial, and complement components, etc.) in a sentence, which can solve the problem of structural ambiguity.

[0141] Word embedding & semantic similarity: to represent words as vectors and calculate the semantic similarity of words based on the vectors, which can solve the problem of word language similarity.

[0142] Text semantic similarity: relying on massive data and deep neural network technology, to realize the ability of semantic similarity calculation between texts, which can solve the problem of text semantic similarity.

[0143] Taking machine translation as an example, machine translation research translates text or speech from one natural language to another natural language by computer program. The compressed neural network obtained by the present application can realize the above NLP tasks.

[0144] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the related terms and concepts related to neural networks involved in the embodiments of the present application will be introduced first.

[0145] (1) Object recognition, using image processing and machine learning, computer graphics and other related methods to determine the category of image objects.

[0146] (2) Neural network

[0147] The neural network can be composed of neural units, and the neural unit can be an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0148]

[0149] where s = 1, 2, … n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. The neural network is a network formed by connecting a plurality of the above single neural units, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of a plurality of neural units.

[0150] (3) Deep neural network

[0151] Deep neural network (DNN) can be understood as a neural network with many hidden layers. Here, "many" has no special measurement standard. The multi-layer neural network and deep neural network commonly referred to are essentially the same thing. From the division of DNN according to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, and the last layer is the output layer. The number of layers in between is the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer. Although DNN looks very complex, in terms of the work of each layer, it is actually not complex. Simply put, it is as follows: wherein, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer only performs the following simple operation on the input vector to obtain the output vector Since the number of layers of DNN is large, the number of coefficients W and offset vectors is also large. Then, how are the specific parameters defined in DNN? First, let's look at the definition of the coefficient W. Take a three-layer DNN as an example: the linear coefficient of the fourth neuron in the second layer to the second neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficient of the k-th neuron in the L-1-th layer to the j-th neuron in the L-th layer is defined as Note that the input layer is without W parameters. In deep neural networks, more hidden layers allow the network to better capture the complexity of real-world situations. In theory, the more parameters a model has, the higher its complexity, and the greater its "capacity" to learn more complex tasks.

[0152] (4) Convolutional Neural Network (CNN) is a deep neural network with convolutional structure. CNN contains a feature extractor composed of convolutional layers and subsampling layers. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolutional feature plane. The convolutional layer refers to the neuron layer in the CNN that performs convolution processing on the input signal. In the convolutional layer of the CNN, a neuron can only be connected to part of the adjacent layer neurons. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units of the same feature plane share weights, and the shared weights are the convolution kernel. Shared weights can be understood as the way of extracting image information regardless of the location. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. That is, the image information learned in one part can also be used in another part. Therefore, for all positions on the image, we can use the same learned image information. In the same convolutional layer, multiple convolution kernels can be used to extract different image information. Generally, the more convolution kernels, the more image information reflected by the convolution operation.

[0153] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning in the training process of the CNN. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the CNN and to reduce the risk of overfitting.

[0154] (5) Backpropagation algorithm

[0155] The CNN can use the backpropagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward propagation of the input signal until the output will produce an error loss, and the error loss information is propagated backward to update the parameters in the initial super-resolution model, so as to make the error loss converge. The backpropagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0156] (6) Loss function

[0157] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0158] (7) Model quantization:

[0159] In the field of artificial intelligence model acceleration, quantization refers to compression techniques that use a bit width of less than 32 bits to represent weight data and feature map data in neural networks. The lower the compressed bit width, the more memory space, latency, and power consumption can be saved in actual deployment. Model quantization is generally divided into two types: post-trained quantization (PTQ) and quantization-aware training (QAT). The former only requires a small amount of unlabeled calibration dataset to complete the quantization process of a pre-trained floating-point model and is often used for compression with a higher bit width. The latter requires a complete dataset for training, simulating quantization operations during training so that the quantized model can further converge to the optimum. It is often used for accuracy recovery after the quantized model has suffered significant losses.

[0160] (8) Peak signal-to-noise ratio (PSNR):

[0161] Peak signal-to-noise ratio (PSNR) is an engineering term representing the ratio of the maximum possible power of a signal to the power of destructive noise that affects the accuracy of its representation. It is frequently used as a measure of signal reconstruction quality in fields such as image processing, and is typically defined simply by mean square error. Generally, the higher the PSNR, the smaller the gap between the representation and the true value.

[0162] Figure 6 The present application provides a hardware structure for a chip, which includes a neural network processor (NPU50). This chip can be configured as follows: Figure 2The execution device 510 shown is configured to perform the computational work of the computational module 511. The chip can also be configured to be mounted in a training device 520 shown to perform the training work of the training device 520 and output the target model / rule. Figure 2 The execution device 510 shown is configured to perform the computational work of the computational module 511. The chip can also be configured to be mounted in a training device 520 shown to perform the training work of the training device 520 and output the target model / rule.

[0163] The neural network processor NPU 50 is mounted as a co-processor on a host central processing unit (CPU) and is assigned tasks by the host CPU. The core part of the NPU is an operation circuit 503, and a controller 504 controls the operation circuit 503 to extract data in a memory (weight memory or input memory) and perform operations.

[0164] In some implementations, the operation circuit 503 internally includes a plurality of processing engines (PEs). In some implementations, the operation circuit 503 is a two-dimensional systolic array. The operation circuit 503 can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 503 is a general-purpose matrix processor.

[0165] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of the matrix B from the weight memory 502 and buffers it on each PE of the operation circuit. The operation circuit takes the data of the matrix A from the input memory 501 and performs matrix operations with the matrix B to obtain partial results or final results of the matrix, which are saved in an accumulator 508.

[0166] The vector computation unit 507 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computation unit 507 can be used for network computation of non-convolution / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0167] In some implementations, the vector computation unit 507 can store the processed output vector to the unified buffer 506. For example, the vector computation unit 507 can apply a non-linear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the operation circuit 503, such as for use in a subsequent layer in a neural network.

[0168] The unified memory 506 is used to store input data and output data.

[0169] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506, the weight memory 502 and the external memory by the direct memory access controller (DMAC) 505.

[0170] The bus interface unit (BIU) 510 is used to realize the interaction among the main CPU, the DMAC and the instruction fetch memory 509 through a bus.

[0171] The instruction fetch memory 509 connected with the controller 504 is used to store the instructions used by the controller 504.

[0172] The controller 504 is used to call the instructions cached in the instruction fetch memory 509 to realize the control of the working process of the operation accelerator.

[0173] Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memories.

[0174] The neural network model (especially the model for realizing the visual related task) needs a large amount of memory space and computing resources in actual operation, which makes it very difficult to deploy them on mobile devices.

[0175] In order to improve the running efficiency, various methods are used to compress the size of the model, such as network pruning, model quantization, lightweight architecture design and knowledge distillation. Among these methods, model quantization is a relatively better technology for existing artificial intelligence acceleration chips, because these chips usually focus on low-precision calculation, so that the time delay, memory occupation and power consumption of model inference can be significantly reduced. However, the accuracy of the compressed model obtained by the model quantization method in the prior art is reduced.

[0176] By setting different bit widths for the weights and feature maps of the model, the cause of the decrease in the quantization accuracy of the underlying visual model can be determined. Taking the super-resolution model as an example, when only the weights of the model are quantized to 6 bits, the performance of the model only decreases slightly (the PSNR index decreases by 0.059, and the SSIM index decreases by 0.002) compared with the full-precision model. In contrast, when only the feature maps are quantized, the accuracy of the model decreases significantly, which indicates that quantizing the features can greatly reduce the performance of the low-precision model.

[0177] Further visualization of the distribution of the feature maps can be used to analyze the specific reasons for the serious performance decline caused by quantization. Three reasons can be summarized as follows:

[0178] (1) Long tail: The distribution of all feature maps is dense in the middle and sparse at both ends, and most of the values are in a very small range, but the range of the entire feature map is very large, for example, as shown in the feature map distribution of body.11.conv1 represented by (b) in FIG. 11, the range after the second input sample is [-232, 210], and 99.16% of the values are between [-30, 25]. For a common uniform quantizer, this very serious long-tail distribution will cause a huge quantization loss in the middle dense area, which will seriously affect the performance. Figure 8A

[0179] (2) Asymmetry: Figure 8A The skew in the figure indicates the skewness, which is an index for measuring the asymmetry of the probability distribution around its mean. For an asymmetric distribution, negative skewness indicates that the long tail is on the left of the distribution, positive skewness indicates that the long tail is on the right of the distribution, and skewness of 0 indicates that the distribution is symmetric at both ends. As can be seen from FIG. 12, the skewness of the feature maps shows that they are very asymmetric at both ends of the distribution, for example, the skewness of the feature maps of body.0.conv1 under two different input samples in FIG. 13 is 3.78 and 1.73, which is not friendly to zero-point calibration using traditional quantization methods. Figure 8A Figure 8A

[0180] (3) High dynamic range: Due to the absence of batch normalization layers, the range of the feature maps of the super-resolution model has very high flexibility. For example, for the body.31.conv1 layer, the range of the output feature maps caused by the second input sample is 2.19 times that of the first sample. High dynamic range means that the optimal quantization value is different for each sample, so it is very challenging to determine an optimal cutoff value that can adapt to the entire super-resolution dataset.

[0181] ​​​In summary, the reason for the sharp decline in the quantization performance of the model is the long-tailed, asymmetric and high dynamic feature map distribution, which leads to the fact that the post-training quantization scheme of the previous high-level visual model cannot be directly migrated to this task. In order to solve this problem, the embodiment of the present application proposes a model quantization method, which can obtain more suitable truncation intervals (that is, upper threshold and lower threshold, or can be called upper boundary threshold and lower boundary threshold) to cope with the long tail and asymmetry of the two characteristics, and then the truncation values can be fine-tuned in a fine-grained manner to drive the quantization model to find the optimal point in the high dynamic feature map change.

[0182] With reference to Figure 7 The model quantization method provided by the embodiment of the present application can include:

[0183] 701、obtain a first feature map, the first feature map being a feature map output by a first intermediate layer of a neural network, the first feature map including a plurality of first feature points, and the neural network being a floating point model.

[0184] The first feature map is a feature map output by an intermediate network layer (for example, a first intermediate layer) of a floating point neural network. The first feature map can include a plurality of first feature points. The embodiment of the present application is mainly used to determine the quantization process of the feature map output by the intermediate network layer.

[0185] First, the basic steps related to model quantization are described as follows:

[0186] Given a floating point tensor x, which needs to be quantized to N bits, set the upper and lower boundaries of the quantization of this tensor as l and u. The quantization process usually includes three steps, as shown in the following formula, 1) truncate the floating point tensor x to [l, u], and set it as x c ; 2) map the floating point x c to the range of [0, 2 N -1], and set it as x int ; 3) reconstruct the floating point tensor from x int , and obtain x q .

[0187] x c = Clamp (x, l, u) ;

[0188]

[0189]

[0190] where Clamp(x, l, u) = min(max(x, l), u), which truncates the input within the given thresholds, outputs the lower threshold if the value is less than the lower threshold, outputs the upper threshold if the value is greater than the upper threshold, and outputs the value itself if the value is within the thresholds. Round() is a rounding function, which outputs the nearest integer to the input. Generally, the goal of post-training quantization is to obtain the upper and lower truncation threshold parameters of each layer weight and feature map without modifying the original floating-point weight, which are set as (l w , u w ) and (l a , u a ) respectively. The purpose of the present application is to quickly obtain the optimal upper and lower truncation thresholds for each layer weight and feature map using only a small number of unlabeled calibration images under a given quantization bit width.

[0191] 702、According to the numerical distribution of the plurality of first feature points, a first truncation interval satisfying a preset condition is determined; the first truncation interval includes a first upper boundary threshold and a first lower boundary threshold, wherein, when quantizing the feature map output by the first intermediate layer, the numerical value of the first feature point less than the first lower boundary threshold is quantized to the first lower boundary threshold, and the numerical value of the first feature point greater than the first upper boundary threshold is quantized to the first upper boundary threshold; the preset condition includes that the numerical distribution density of the feature points within the numerical truncation interval is greater than the numerical distribution density of the feature points outside the numerical truncation interval.

[0192] In a possible implementation, the preset condition further includes that the number of feature points within the numerical truncation interval and the proportion of the number of feature points included in the feature map are greater than a first threshold.

[0193] A system architecture diagram of an embodiment of the present application can be as shown in Figure 8B The optimization of post-training quantization of the model can include two steps, the first step is density-based double-end truncation, by continuously comparing the distribution densities of the upper and lower ends, the outlying values in the floating-point parameters are removed based on considering the long-tail and asymmetric distribution of the feature map, to obtain a floating-point model with relatively rough upper and lower boundaries.

[0194] Through analysis of the feature map, it is known that asymmetric and long-tail distribution are the second main reason affecting the quantization performance of the model. This module fully considers this characteristic, based on the priori cognition that the lower the density, the lower the importance, by comparing the densities of the upper threshold and the lower threshold, the end with lower density is truncated, and the end with higher density is retained, and then the iteration is continued until the proportion of the density within the threshold range to the entire feature map density is lower than a certain threshold, and the iteration process is stopped, to obtain the current optimal truncation parameter.

[0195] In a possible implementation, the numerical range of the plurality of first feature points can be divided into a plurality of numerical intervals according to numerical size; a numerical interval with a smaller numerical distribution density is sequentially determined from both sides of the plurality of numerical intervals to an inside as an edge numerical interval, until the number of first feature points in a numerical interval other than the edge numerical interval in the plurality of numerical intervals and the proportion of the number of the plurality of first feature points are less than a second threshold value, and the numerical interval is determined as the first truncation interval.

[0196] The distribution of the feature map output by the intermediate layer of the model is usually shown as dense in the middle and sparse at both ends, so the dense region is far away from the original boundary, which is very unfriendly to model quantization, especially for low bits. Therefore, the embodiment of the present application proposes to cut off the outliers of the feature map based on the density of the double-end truncation to help reduce the distribution to an effective range.

[0197] Unlike the single-end truncation of the high-level semantic model using the ReLU function, the embodiment of the present application fully considers the asymmetric distribution of the numerical values of the feature points in the feature map output by the intermediate layer of the model. Taking the range truncation of a certain layer as an example, first, the range of the original feature map x is divided into N equal parts according to the maximum and minimum values of the original feature map x:

[0198]

[0199]

[0200] where H(p) represents the density of position p. Based on the assumption that the lower the distribution density on the boundary, the lower the importance to the whole, the goal of the present technology is to keep the area of the part with the highest density as much as possible. As shown in Figure 8B , H(p) is usually very small at both ends and very large in the middle. Therefore, in order to obtain a suitable upper end truncation threshold and a lower end truncation threshold, the embodiment of the present application can start searching from the original two points, which are called l a and u a . The distribution density of the two threshold positions is compared constantly, and the boundary point with lower density is cut off to move it closer to the middle. This process can be represented by the following formula:

[0201]

[0202] where t represents the current iteration step number, and the termination condition for iteration of a batch sample is to stop when the proportion of the sum of the densities in the truncation threshold to the entire distribution density is lower than a certain threshold (M%).

[0203] In this way, the upper end truncation threshold and the lower end truncation threshold are iterated in a form of distribution density comparison, and the boundary part with lower importance (lower density) is truncated, so as to solve the quantization loss caused by the long-tail distribution and the asymmetric distribution.

[0204] In the embodiments of the present application, different truncation intervals can be determined based on the numerical distribution of the feature map itself for the output feature map of different intermediate layers in the neural network. Taking a second intermediate layer different from the first intermediate layer as an example:

[0205] In a possible implementation, a third feature map can also be obtained, the third feature map being a feature map output by a second intermediate layer of a neural network, the third feature map including a plurality of third feature points, the neural network being a floating point model; a third truncation interval satisfying the preset condition is determined according to a numerical distribution of the plurality of third feature points; the third truncation interval includes a third upper boundary threshold and a third lower boundary threshold, wherein, when quantizing the feature map output by the second intermediate layer, a numerical value of a third feature point less than the third lower boundary threshold is quantized to the third lower boundary threshold, and a numerical value of a third feature point greater than the third upper boundary threshold is quantized to the third upper boundary threshold.

[0206] In a possible implementation, the first feature map is a feature map output by a first intermediate layer of a neural network when processing a first batch of training samples; when a next batch of training samples is input, a second feature map can be obtained, the second feature map being a feature map output by the first intermediate layer of the neural network when processing a second batch of training samples, the second feature map including a plurality of second feature points; a second truncation interval satisfying the preset condition is determined according to a numerical distribution of the plurality of second feature points; the second truncation interval includes a second upper boundary threshold and a second lower boundary threshold; the first truncation interval is updated according to the second truncation interval to obtain a third truncation interval.

[0207] In a possible implementation, the first truncation interval can be updated by an exponential moving average according to the second truncation interval.

[0208] After obtaining the most suitable upper end and lower end truncation thresholds of the current calibration sample, the next batch of calibration samples is input, and the global upper end and lower end threshold parameter values are also updated by using the exponential moving average, which can be expressed by the following formula:

[0209]

[0210]

[0211] where β is a hyperparameter of weight decay, and and represents the truncation value of the current batch, and a specific algorithm flowchart is shown as Figure 9

[0212] Pixel-aware calibration can then be performed, and the model is quantized based on the floating-point parameters obtained in the first step. Then, the full-precision model is used to supervise the low-bit quantization model, and the quantization parameters are further fine-tuned on a small calibration dataset, so that they further adapt to the highly dynamic feature map distribution changes during the fine-tuning process.

[0213] Specifically, on the basis of the first step, further consideration is given to the fact that the high dynamic change of the feature map is one of the main factors affecting the quantization performance. The floating-point model is used as a supervision model to guide the quantization model to further fine-tune the threshold parameters of the quantization model through the intermediate features and the final output. In this fine-tuning process, the threshold parameters of the quantization model will continuously adapt to the highly dynamic change of the feature map, and find the optimal point under the supervision of the floating-point model, thereby obtaining a high-precision quantization model.

[0214] In a possible implementation, the first truncation interval is used to quantize the neural network to obtain a quantized neural network; the first output and the second output can be obtained, the first output being an output of an intermediate layer or an output layer of the neural network when processing a training sample, and the second output being an output of an intermediate layer or an output layer of the quantized neural network when processing the training sample; a loss is determined according to the first output and the second output, and the first truncation interval is updated according to the loss.

[0215] For example, the first output and the second output can be outputs of an intermediate layer of the neural network when processing a training sample, or the first output and the second output can be outputs of an output layer of the neural network when processing a training sample.

[0216] In a possible implementation, when constructing the loss, the L2 norm of the first output and the second output can be calculated respectively to obtain a processed first output and a processed second output; and the loss is determined according to the mean square error between the processed first output and the processed second output.

[0217] ​In the coarse adjustment technique based on the density-based double-end truncation, initial lower-end and upper-end truncation thresholds can be obtained for the weights and feature maps of each layer of the model, and in this step, the weights and feature maps are not quantized and always remain in the form of a floating-point model. Then the embodiment of the application further proposes a pixel perception calibration method to fine-tune these truncation parameters to fit the high dynamic feature maps of different samples under a given bit width setting. By using unmarked calibration images and a full-precision pre-trained model, the feature maps of the model output and the intermediate layer output can be obtained to provide rich supervision information for the quantized model and be used to construct a data set for fine-tuning. The input and label pair of the i-th sample can be expressed as:

[0218]

[0219] wherein O represents the output feature map of the b-th residual module, O i represents the output of the floating-point model for the i-th sample, and thus a small fine-tuning data set with only 100 input and label pairs can be obtained. Since the number of calibration images is limited, the fine-tuning of this method is only the truncation parameters, so that the number of parameters and the number of samples are similar, thereby avoiding overfitting. The supervision of the floating-point model on the quantized model includes two parts, one part is the output of the model, and the other part is the feature map. For the output, the traditional L1 loss can be adopted:

[0220]

[0221] wherein ||·||1 represents the L1 norm, H o , W o and C o represent the height, width and channel number of the output image, O and O q represent the output results of the floating-point model and the quantized model for the same set of input samples. For the supervision of the feature map, a pixel migration loss is proposed to calculate the distance between the outputs of the intermediate layers of the floating-point model and the quantized model. This technique first calculates an L2 norm of the feature maps of the two models:

[0222]

[0223] wherein F i and F qi represent the outputs of the i-th residual block of the floating-point model and the quantized model, respectively. Then the mean square error of the two feature maps of all modules is calculated, and the expression of the pixel migration loss is as follows:

[0224]

[0225] wherein H i , W i and Ci where H, W and C represent the height, width and channel number of the output feature map of the i-th residual module, and B represents the number of modules. Then the final total loss function can be obtained as:

[0226] L PaC = L o + λL pt ;

[0227] where λ is a hyper-parameter used to balance the two losses, which can be optionally set to 5. In the case of fully considering the total loss minimization of the reconstruction loss and the cumulative error of quantization, the quantization model tends to imitate the full-precision model and try to find the most suitable threshold parameter for the high dynamic distribution.

[0228] In one possible implementation, the first threshold interval can be updated according to the loss while keeping the threshold interval corresponding to the weight parameter of the neural network unchanged.

[0229] In order to stabilize the fine-tuning process, the clipping parameters of the weight and the feature map can be iteratively optimized instead of being fine-tuned together. As shown in FIG. 5, the threshold parameter of the feature map is first frozen, and the threshold parameter of the weight is fine-tuned with the total loss, where the gradient can be represented as: Figure 8B

[0230]

[0231]

[0232] where is the threshold parameter of the i-th layer weight. Optionally, for the gradient can be approximated as 1 here. Then the threshold parameter of the weight is frozen, and the parameter of the feature map is fine-tuned with the same loss function. The calculation of the gradient is the same as that of the weight. This iterative optimization cycle is repeated until it reaches the calibrated round, and during the entire calibration process, the original weight parameter does not participate in the update.

[0233] ​The embodiment of the present application provides a model quantization method, which comprises the following steps: acquiring a first feature map, wherein the first feature map is a feature map output by a first intermediate layer of a neural network, the first feature map comprises a plurality of first feature points, and the neural network is a floating-point model; determining a first truncation interval that satisfies a preset condition according to a numerical distribution of the plurality of first feature points; the first truncation interval comprises a first upper boundary threshold and a first lower boundary threshold, wherein when the feature map output by the first intermediate layer is quantized, the numerical value of the first feature point smaller than the first lower boundary threshold is quantized as the first lower boundary threshold, and the numerical value of the first feature point greater than the first upper boundary threshold is quantized as the first upper boundary threshold; and the preset condition comprises that the numerical distribution density of the feature points in the numerical truncation interval is greater than the numerical distribution density of the feature points outside the numerical truncation interval. The present application uses an upper end truncation threshold and a lower end truncation threshold to represent the parameter setting of quantization, instead of the zero position and the range size commonly used in the previous scheme, uses the density-based double-end cutting on the floating-point model, removes outliers in the long-tail distribution, can adapt to the asymmetric distribution trend, and further improves the precision of the quantized model.

[0234] In addition, the strategy of using pixel perception calibration on the model is used to appropriately fine-tune the quantization parameters, so that the quantized model can better adapt to the high dynamic feature change.

[0235] The present application only needs a small amount of unlabeled calibration data, and the quantized model can be obtained in a few minutes. Compared with the quantized perception training, the embodiment of the present application does not need a complete training data set and high-cost training server resources, greatly reduces the cost required by the model quantization process, and can obtain the structure and parameters of the quantized model in a short time, greatly speeds up the efficiency of the model online deployment.

[0236] The structural block diagram of one embodiment of the embodiment of the present application is shown in Figure 10 The modules and main steps in the example are described in detail below.

[0237] After a given trained floating-point model, the floating-point weights and feature map distribution ranges of each layer are truncated using unlabeled calibration pictures. After each sample batch is input, the density of the upper end threshold and the lower end threshold is compared constantly, the most suitable parameters are obtained through iterative optimization, and then the global upper end threshold and lower end threshold parameters are obtained by using the exponential moving average. In the case of fully considering the long tail and asymmetric feature map, outliers in the distribution are cut off, and a floating-point model with appropriate truncation threshold is obtained.

[0238] On the basis of the first step, the model is quantized under the condition of a given bit width. A small fine-tuning dataset is constructed using unlabeled pictures and the original floating-point model, and then the intermediate feature maps and the final output of the floating-point model are used as a supervision signal to guide the threshold parameters of the quantized model to be fine-tuned, so as to further adapt to the distribution changes of the high dynamic feature maps of the underlying visual model, and finally obtain the quantized model.

[0239] On the given training test set, the embodiments of the present application are compared with the following benchmark algorithms, and have the following beneficial effects.

[0240] Benchmark algorithm 1: the results of the floating-point model and the Bicubic algorithm come from EDSR in the paper "Enhanced deep residual networks for single image super-resolution" and SRResNet in the paper "Photorealistic single image super-resolution using a generative adversarial network", respectively.

[0241] Benchmark algorithm 2: commercial deployment tools, including OpenVINO, TensorRT, SNPE.

[0242] Benchmark algorithm 3: post-training quantization algorithm for high-level semantics, including the MSE algorithm in the paper "Bridging the accuracy gap for 2-bit quantized neural networks", the percentile algorithm in the paper "Fully quantized network for object detection", and the MinMax algorithm in the paper "Quantization and training of neural networks for efficient integer-arithmetic-only inference".

[0243] Benchmark algorithm 4: quantization-aware training algorithm for underlying visual models, including the PAMS algorithm in the paper "Pams: Quantized super-resolution via parameterized max scale" and the FQSR algorithm in the paper "Fully quantized image super-resolution networks".

[0244] Beneficial effect 1: Better objective indicators on EDSR model. As shown in Table 1, compared with existing methods, the embodiments of the present application have better PSNR / SSIM (the larger the better) on open source datasets Set5, Set14, BSDS100 and Urban100 and EDSR model. For example, when the model is quantized to 4-bit, the precision error of the EDSRx4 model quantized by the embodiments of the present application is only 1.28dB on Set5, far exceeding other solutions.

[0245] Beneficial effect 2: Better objective indicators on SRResNet model. As shown in Table 2, compared with existing methods, the embodiments of the present application have better PSNR / SSIM (the larger the better) on open source datasets Set5, Set14, BSDS100 and Urban100 and SRResNet model. For example, when the model is quantized to 4-bit, the precision error of the SRResNetx4 model quantized by the embodiments of the present application is only 1.09dB on Set5, far exceeding other solutions.

[0246] Beneficial effect 3: Better objective indicators than quantization-aware training solutions. As shown in Table 3, compared with existing methods, the embodiments of the present application have better PSNR / SSIM (the larger the better) on open source datasets Set5, Set14, BSDS100 and Urban100 and EDSR model.

[0247] Beneficial effect 4: Better visual effects. As shown in Table 4, compared with other methods, the method of the embodiments of the present application is closer to the floating-point model in detail preservation, and can obtain better subjective and objective effects. Figure 11

[0248] Table 1 Comparison of objective indicators (PSNR / SSIM) of post-training quantization methods on EDSR

[0249]

[0250] Table 2 Comparison of objective indicators (PSNR / SSIM) of post-training quantization methods on SRResNet

[0251]

[0252] Table 3 Comparison of objective indicators (PSNR / SSIM) of quantization-aware training methods on EDSR

[0253]

[0254] Reference Figure 12 , Figure 12 ​A structure of a model quantization device provided in an embodiment of the present application is shown in FIG. 12. As shown in FIG. 12, the device 1200 includes: Figure 12

[0255] An acquisition module 1201 is configured to acquire a first feature map, where the first feature map is a feature map output by a first intermediate layer of a neural network, the first feature map includes a plurality of first feature points, and the neural network is a floating-point model.

[0256] For details of the acquisition module 1201, refer to the description of step 701 in the above embodiments, which will not be repeated here.

[0257] A processing module 1202 is configured to determine a first truncation interval that satisfies a preset condition according to a numerical distribution of the plurality of first feature points, where the first truncation interval includes a first upper boundary threshold and a first lower boundary threshold, when quantizing the feature map output by the first intermediate layer, a numerical value of a first feature point less than the first lower boundary threshold is quantized to the first lower boundary threshold, and a numerical value of a first feature point greater than the first upper boundary threshold is quantized to the first upper boundary threshold, and the preset condition includes that a numerical distribution density of a feature point in the numerical truncation interval is greater than a numerical distribution density of a feature point outside the numerical truncation interval.

[0258] For details of the processing module 1202, refer to the description of step 702 in the above embodiments, which will not be repeated here.

[0259] In a possible implementation, the preset condition further includes:

[0260] A number of feature points in the numerical truncation interval and a proportion of the number of feature points included in the feature map are greater than a first threshold.

[0261] In a possible implementation, the processing module is specifically configured to:

[0262] divide a numerical range of the plurality of first feature points into a plurality of numerical intervals according to numerical values;

[0263] determine, from both sides of the plurality of numerical intervals to the inside in sequence, a numerical interval with a smaller numerical distribution density as an edge numerical interval, until a number of first feature points in other numerical intervals except the edge numerical interval in the plurality of numerical intervals and a proportion of the number of the plurality of first feature points are less than a second threshold, and determine the other numerical intervals as the first truncation interval.

[0264] In a possible implementation, the first feature map is a feature map output by a first intermediate layer when the neural network processes a first batch of training samples.

[0265] ​The acquisition module is further configured to:

[0266] acquire a second feature map, the second feature map being a feature map output by the first intermediate layer when the neural network processes a second batch of training samples, the second feature map including a plurality of second feature points;

[0267] The processing module is further configured to:

[0268] determine, according to a numerical distribution of the plurality of second feature points, a second clipping interval that satisfies the preset condition, the second clipping interval including a second upper boundary threshold and a second lower boundary threshold;

[0269] update the first clipping interval according to the second clipping interval to obtain a third clipping interval.

[0270] In a possible implementation, the processing module is specifically configured to:

[0271] update the first clipping interval by exponential moving average according to the second clipping interval.

[0272] In a possible implementation, the acquisition module is further configured to:

[0273] acquire a third feature map, the third feature map being a feature map output by a second intermediate layer of a neural network, the third feature map including a plurality of third feature points, and the neural network being a floating-point model;

[0274] The processing module is further configured to:

[0275] determine, according to a numerical distribution of the plurality of third feature points, a third clipping interval that satisfies the preset condition, the third clipping interval including a third upper boundary threshold and a third lower boundary threshold, wherein, when quantizing the feature map output by the second intermediate layer, a numerical value of a third feature point less than the third lower boundary threshold is quantized to the third lower boundary threshold, and a numerical value of a third feature point greater than the third upper boundary threshold is quantized to the third upper boundary threshold.

[0276] In a possible implementation, the first clipping interval is used to quantize the neural network to obtain a quantized neural network, and the acquisition module is further configured to:

[0277] The processing module is further configured to:

[0278] acquire a first output and a second output, the first output being an output of an intermediate layer or an output layer of the neural network when the neural network processes a training sample, and the second output being an output of an intermediate layer or an output layer of the quantized neural network when the quantized neural network processes the training sample;

[0279] According to the first output and the second output, a loss is determined, and the first truncated interval is updated according to the loss.

[0280] In a possible implementation, the training sample is unlabeled data.

[0281] In a possible implementation, the processing module is specifically configured to:

[0282] An L2 norm is calculated for the first output and the second output respectively to obtain a processed first output and a processed second output.

[0283] According to a mean square error between the processed first output and the processed second output, a loss is determined.

[0284] In a possible implementation, the processing module is specifically configured to:

[0285] The first truncated interval is updated according to the loss while keeping the truncated interval corresponding to the weight parameter of the neural network unchanged.

[0286] Next, an execution device provided by an embodiment of the present application is introduced. Please refer to Figure 13 , Figure 13 A structural schematic diagram of the execution device provided by the embodiment of the present application is shown in FIG. 13. The execution device 1300 can be a virtual reality (VR) device, a mobile phone, a tablet, a notebook computer, a smart wearable device, a monitoring device, a server, or the like, which is not limited herein. Specifically, the execution device 1300 includes a receiver 1301, a transmitter 1302, a processor 1303, and a memory 1304 (wherein the number of processors 1303 in the execution device 1300 can be one or more, and one processor is taken as an example in the embodiment), wherein the processor 1303 can include an application processor 13031 and a communication processor 13032. In some embodiments of the present application, the receiver 1301, the transmitter 1302, the processor 1303, and the memory 1304 can be connected through a bus or other means. Figure 13

[0287] The memory 1304 can include a read-only memory and a random access memory, and provide the processor 1303 with instructions and data. A part of the memory 1304 can also include a non-volatile random access memory (NVRAM). The memory 1304 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or expanded sets thereof, wherein the operation instructions can include various operation instructions for implementing various operations.

[0288] ​The processor 1303 controls the operation of the execution device. In a specific application, various components of the execution device are coupled together by a bus system, which can include a data bus, a power bus, a control bus, and a state signal bus, etc. However, for the sake of clarity, various buses are referred to as a bus system in the figure.

[0289] The method disclosed in the embodiments of the present application can be applied in the processor 1303 or implemented by the processor 1303. The processor 1303 can be an integrated circuit chip having a signal processing capability. In the implementation process, the steps of the above method can be completed by hardware integrated logic circuit or software form instructions in the processor 1303. The processor 1303 described above can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The processor 1303 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 1304, and the processor 1303 reads the information in the memory 1304, and combines the hardware to run the above-mentioned Figure 7 The corresponding method obtains a quantized model.

[0290] The receiver 1301 can be used to receive input digital or character information, and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1302 can be used to output digital or character information through the first interface; the transmitter 1302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1302 can also include a display device such as a display screen.

[0291] The embodiments of the present application also provide a training device, please refer to Figure 14 , Figure 14is a structural schematic diagram of a training device provided by an embodiment of the present application. Specifically, the training device 1400 is implemented by one or more servers. The training device 1400 can have a large difference due to different configurations or performances, and can include one or more central processing units (CPUs) 1414 (for example, one or more processors) and a memory 1432, one or more storage media 1430 (for example, one or more mass storage devices) storing an application program 1442 or data 1444. The memory 1432 and the storage media 1430 can be temporary storage or persistent storage. The program stored in the storage media 1430 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the training device. Further, the central processing unit 1414 can be configured to communicate with the storage media 1430 and execute the series of instruction operations in the storage media 1430 on the training device 1400.

[0292] The training device 1400 can further include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input / output interfaces 1458, or one or more operating systems 1441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and the like.

[0293] In an embodiment of the present application, the central processing unit 1414 is configured to execute the series of instruction operations in the training device 1400. Figure 7 The model quantization method described in the corresponding embodiment.

[0294] An embodiment of the present application further provides a computer program product, which, when running on a computer, causes the computer to perform the steps executed by the execution device described above, or causes the computer to perform the steps executed by the training device described above.

[0295] An embodiment of the present application further provides a computer readable storage medium, which stores a program for signal processing, and when running on a computer, causes the computer to perform the steps executed by the execution device described above, or causes the computer to perform the steps executed by the training device described above.

[0296] The execution device, the training device or the terminal device provided by the embodiments of the present application can be a chip, which includes a processing unit, for example, a processor, and a communication unit, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the execution device executes the model quantization method described in the above embodiments, or so that the chip in the training device executes the model quantization method described in the above embodiments. Alternatively, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0297] Specifically, refer to Figure 15 , Figure 15 A structural diagram of the chip provided by the embodiments of the present application is shown in FIG. 15. The chip can be a neural network processor NPU 1500, which is mounted on a host CPU as a coprocessor and is assigned tasks by the host CPU. The core part of the NPU is an operation circuit 1503, which extracts matrix data in a memory and performs multiplication operation under the control of a controller 1504.

[0298] In some implementations, the operation circuit 1503 internally includes a plurality of processing units (PEs). In some implementations, the operation circuit 1503 is a two-dimensional systolic array. The operation circuit 1503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1503 is a general-purpose matrix processor.

[0299] For example, assume that there are an input matrix A, a weight matrix B and an output matrix C. The operation circuit takes corresponding data of the matrix B from the weight memory 1502 and buffers them on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1501 and performs matrix operation with the matrix B, and the partial result or final result of the obtained matrix is saved in an accumulator 1508.

[0300] The unified memory 1506 is used to store input data and output data. The weight data is transferred to the weight memory 1502 through a Direct Memory Access Controller (DMAC) 1505. The input data is also transferred to the unified memory 1506 through the DMAC.

[0301] The BIU is a Bus Interface Unit 1510 for the interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1509.

[0302] The BIU 1510 is used for the instruction fetch buffer 1509 to fetch instructions from the external memory, and is also used for the memory access controller 1505 to fetch the original data of the input matrix A or the weight matrix B from the external memory.

[0303] The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1506, or to transfer the weight data to the weight memory 1502, or to transfer the input data to the input memory 1501.

[0304] The vector calculation unit 1507 includes a plurality of operation processing units, which further process the output of the operation circuit 1503 as needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / full connection layer network calculation in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0305] In some implementations, the vector calculation unit 1507 can store the processed output vector to the unified memory 1506. For example, the vector calculation unit 1507 can apply a linear function; or, a non-linear function to the output of the operation circuit 1503, such as linear interpolation on the feature planes extracted by the convolutional layer, and further, a vector of accumulated values to generate activation values. In some implementations, the vector calculation unit 1507 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as activation input to the operation circuit 1503, such as for use in subsequent layers in the neural network.

[0306] The controller 1504 is connected to the instruction fetch buffer 1509, which is used to store instructions used by the controller 1504;

[0307] The unified memory 1506, the input memory 1501, the weight memory 1502, and the instruction memory 1509 are on-chip memories. The external memory is private to the NPU hardware architecture.

[0308] Any processor mentioned in the above can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above programs.

[0309] It should be noted that the above-described apparatus embodiments are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0310] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits, or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, a training device, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0311] In the above embodiments, all or part can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part.

[0312] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A model quantization method, characterized in that, The method comprises: obtaining a first feature map, the first feature map being a feature map output by a first intermediate layer of a neural network, the first feature map comprising a plurality of first feature points, the neural network being a floating-point model, the neural network being applied to a visual perception task, a speech semantics related natural language synthesis task, or an audio-video processing task; determining a first truncation interval satisfying a preset condition according to a numerical distribution of the plurality of first feature points; the first truncation interval comprising a first upper boundary threshold and a first lower boundary threshold, wherein, when quantizing the feature map output by the first intermediate layer, a numerical value of a first feature point less than the first lower boundary threshold is quantized to the first lower boundary threshold, and a numerical value of a first feature point greater than the first upper boundary threshold is quantized to the first upper boundary threshold; the preset condition comprising that a numerical distribution density of feature points within the numerical truncation interval is greater than a numerical distribution density of feature points outside the numerical truncation interval.

2. The method of claim 1, wherein, The preset condition further comprises: a number of feature points within the numerical truncation interval and a proportion of the number of feature points in the feature map are greater than a first threshold.

3. The method according to claim 1 or 2, characterized in that, The determining of the first truncation interval satisfying the preset condition according to the numerical distribution of the plurality of first feature points comprises: dividing a numerical range of the plurality of first feature points into a plurality of numerical intervals according to numerical values; determining, from both sides to the inside of the plurality of numerical intervals, a numerical interval with a smaller numerical distribution density as an edge numerical interval in sequence until a number of first feature points in other numerical intervals except the edge numerical interval in the plurality of numerical intervals and a proportion of the number of the plurality of first feature points are less than a second threshold, and determining the other numerical intervals as the first truncation interval.

4. The method according to claim 1 or 2, characterized in that, The first feature map is a feature map output by the first intermediate layer when the neural network processes a first batch of training samples; The method further comprises: obtaining a second feature map, the second feature map being a feature map output by the first intermediate layer when the neural network processes a second batch of training samples, the second feature map comprising a plurality of second feature points; determining a second truncation interval satisfying the preset condition according to a numerical distribution of the plurality of second feature points; the second truncation interval comprising a second upper boundary threshold and a second lower boundary threshold; updating the first truncation interval according to the second truncation interval to obtain a third truncation interval.

5. The method of claim 4, wherein, The updating of the first truncation interval according to the second truncation interval comprises: updating the first truncation interval by exponential moving average according to the second truncation interval.

6. The method according to claim 1 or 2, characterized in that, The method further comprises: obtaining a third feature map, the third feature map being a feature map output by a second intermediate layer of a neural network, the third feature map comprising a plurality of third feature points, the neural network being a floating-point model; determine a third clipping interval satisfying the preset condition according to the numerical distribution of the third feature points; the third clipping interval comprises a third upper boundary threshold and a third lower boundary threshold, wherein, when quantizing the feature map output by the second intermediate layer, the numerical value of the third feature point less than the third lower boundary threshold is quantized to the third lower boundary threshold, and the numerical value of the third feature point greater than the third upper boundary threshold is quantized to the third upper boundary threshold.

7. The method according to claim 1 or 2, characterized in that, The first clipping interval is used to quantize the neural network to obtain a quantized neural network; the method further comprises: obtaining a first output and a second output, the first output being an output of an intermediate layer or an output layer of the neural network when processing a training sample, and the second output being an output of an intermediate layer or an output layer of the quantized neural network when processing the training sample; determining a loss according to the first output and the second output, and updating the first clipping interval according to the loss.

8. The method of claim 7, wherein, The training sample is unlabeled data.

9. The method of claim 7, wherein, The first output and the second output are intermediate layer output feature maps. The determination of the loss according to the first output and the second output comprises: calculating the L2 norm of the first output and the second output respectively to obtain a processed first output and a processed second output; determining the loss according to the mean square error between the processed first output and the processed second output.

10. The method of claim 7, wherein, The updating of the first clipping interval according to the loss comprises: updating the first clipping interval according to the loss while keeping the clipping interval corresponding to the weight parameter of the neural network unchanged.

11. A model quantization apparatus, comprising: The apparatus comprises: an obtaining module configured to obtain a first feature map, the first feature map being a feature map output by a first intermediate layer of a neural network, the first feature map comprising a plurality of first feature points, the neural network being a floating point model, and the neural network being applied to a visual perception task, a speech semantic related natural language synthesis task, or an audio and video processing task; a processing module configured to determine a first clipping interval satisfying a preset condition according to the numerical distribution of the plurality of first feature points; the first clipping interval comprises a first upper boundary threshold and a first lower boundary threshold, wherein, when quantizing the feature map output by the first intermediate layer, the numerical value of the first feature point less than the first lower boundary threshold is quantized to the first lower boundary threshold, and the numerical value of the first feature point greater than the first upper boundary threshold is quantized to the first upper boundary threshold; the preset condition comprises that the numerical distribution density of the feature points within the numerical clipping interval is greater than the numerical distribution density of the feature points outside the numerical clipping interval.

12. The apparatus of claim 11, wherein, The preset condition further comprises: the number of feature points within the numerical clipping interval and the proportion of the number of feature points in the feature map are greater than a first threshold.

13. The apparatus of claim 11 or 12, wherein, The processing module is specifically configured to: divide the numerical range of the plurality of first feature points into a plurality of numerical intervals according to numerical size; The number of first feature points in the other numerical interval and the proportion of the number of the plurality of first feature points are less than a second threshold value, and the other numerical interval is determined as the first cut-off interval.

14. The apparatus of claim 11 or 12, wherein, The first feature map is a feature map output by the first intermediate layer of the neural network when processing a first batch of training samples. The obtaining module is further configured to: obtain a second feature map, the second feature map being a feature map output by the first intermediate layer of the neural network when processing a second batch of training samples, and the second feature map including a plurality of second feature points; The processing module is further configured to: determine a second cut-off interval satisfying the preset condition according to the numerical distribution of the plurality of second feature points; the second cut-off interval including a second upper boundary threshold value and a second lower boundary threshold value; update the first cut-off interval according to the second cut-off interval to obtain a third cut-off interval.

15. The apparatus of claim 14, wherein, The processing module is specifically configured to: update the first cut-off interval according to the second cut-off interval by exponential moving average.

16. The apparatus of claim 11 or 12, wherein, The obtaining module is further configured to: obtain a third feature map, the third feature map being a feature map output by a second intermediate layer of the neural network, the third feature map including a plurality of third feature points, and the neural network being a floating-point model; The processing module is further configured to: determine a third cut-off interval satisfying the preset condition according to the numerical distribution of the plurality of third feature points; the third cut-off interval including a third upper boundary threshold value and a third lower boundary threshold value, wherein, when quantizing the feature map output by the second intermediate layer, a numerical value of a third feature point less than the third lower boundary threshold value is quantized to the third lower boundary threshold value, and a numerical value of a third feature point greater than the third upper boundary threshold value is quantized to the third upper boundary threshold value.

17. The apparatus of claim 11 or 12, wherein, The first cut-off interval is used to quantize the neural network to obtain a quantized neural network; the obtaining module is further configured to: The processing module is further configured to: obtain a first output and a second output, the first output being an output of an intermediate layer or an output layer of the neural network when processing a training sample, and the second output being an output of an intermediate layer or an output layer of the quantized neural network when processing the training sample; determine a loss according to the first output and the second output, and update the first cut-off interval according to the loss.

18. The apparatus of claim 17, wherein, The training sample is unlabeled data.

19. The apparatus of claim 17, wherein, The processing module is specifically configured to: calculate L2 norms of the first output and the second output respectively to obtain a processed first output and a processed second output; determine a loss according to a mean square error between the processed first output and the processed second output.

20. The apparatus of claim 17, wherein, The processing module is specifically configured to: update the first cut-off interval according to the loss while keeping the cut-off interval corresponding to a weight parameter of the neural network unchanged.

21. A computer storage medium, comprising, The computer storage medium stores one or more instructions which, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1-10.

22. A computer program product, characterised in that, Computer readable instructions are included which, when run on a computer device, cause the computer device to perform the method of any one of claims 1-10.

23. A system comprising at least one processor, at least one memory; the processor, the memory are connected through a communication bus and complete the communication between each other; The at least one memory is configured to store code; The at least one processor is configured to execute the code to perform the method of any one of claims 1-10.

Images