Sparse Convolutional Neural Networks

Abstract

Various implementations of the present disclosure relate to sparse convolutional neural networks. In some implementations, a computer-implemented method includes: quantizing an input feature map to obtain a quantized input feature map; based on the quantized input feature map, determining a sparse mask for an output feature map by a quantized version of a convolutional neural network, the sparse mask indicating the locations of non-zero items in the output feature map; based on the input feature map, determining the non-zero items in the output feature map indicated by the sparse mask by the convolutional neural network.

Description

Background Art

[0001] Currently, many major advances in artificial intelligence come from convolutional neural networks (CNNs) with great depth. Inference on these very large models usually requires a lot of computing resources and consumes a lot of energy. Therefore, how to implement these models in a low-computational-cost and energy-efficient way is a problem that needs to be solved. Summary of the invention

[0002] According to some implementations of the present disclosure, a scheme for reducing the overhead of a convolutional neural network by quantization and sparse convolution is provided. For example, an input feature map can be quantized to obtain a quantized input feature map. Based on the quantized input feature map, a sparse mask for an output feature map is determined by a quantized version of the convolutional neural network, wherein the binary sparse mask indicates the location of non-zero items in the output feature map. Based on the input feature map, the non-zero items indicated by the sparse mask in the output feature map are determined by the convolutional neural network.

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 A block diagram showing a computing device capable of implementing various implementations of the present disclosure is shown;

[0005] Figure 2 A schematic diagram showing the architecture of a convolutional neural network according to some implementations of the present disclosure;

[0006] Figure 3 A schematic diagram showing a quantization operation according to some implementations of the present disclosure is shown;

[0007] Figure 4 A schematic diagram showing a method for determining a sparse mask according to some implementations of the present disclosure is shown;

[0008] Figure 5 A schematic diagram showing an encoding method according to some implementations of the present disclosure; and

[0009] Figure 6 A flowchart of a method for a convolutional neural network according to some implementations of the present disclosure is shown.

[0010] In these drawings, the same or similar reference symbols are used to designate the same or similar elements. DETAILED DESCRIPTION

[0011] The present disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and thus implement the present disclosure, and do not imply any limitation on the scope of the present subject matter.

[0012] As used herein, the term "including" and variations thereof are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "based at least in part on". The terms "an implementation" and "an implementation" are to be interpreted as "at least one implementation". The term "another implementation" is to be interpreted as "at least one other implementation". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0013] The basic principles and several example implementations of the present disclosure are explained below with reference to the accompanying drawings. Figure 1 1 is a block diagram of a computing device 100 capable of implementing multiple implementations of the present disclosure. It should be understood that Figure 1 The computing device 100 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described in the present disclosure. Figure 1 As shown, computing device 100 comprises a computing device in the form of a general purpose computing device 100. Components of computing device 100 may include, but are not limited to, one or more processors or processing units 110, memory 120, storage device 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160.

[0014] In some implementations, the computing device 100 can be implemented as various user terminals or service terminals with computing capabilities. The service terminal can be a server, a large computing device, etc. provided by various service providers. The user terminal is such as any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a site, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. It is also foreseeable that the computing device 100 can support any type of interface for the user (such as a "wearable" circuit, etc.).

[0015] Processing unit 110 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 120. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device 100. Processing unit 110 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0016] The computing device 100 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device 100, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 120 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The memory 120 may include a text generation module 122, which are program modules configured to perform the functions of the various implementations described herein. The text generation module 122 may be accessed and run by the processing unit 110 to implement the corresponding functions.

[0017] Storage device 130 may be a removable or non-removable medium and may include machine-readable media that can be used to store information and / or data and can be accessed within computing device 100. Computing device 100 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 1 As shown in , a disk drive for reading or writing from a removable, nonvolatile disk and an optical drive for reading or writing from a removable, nonvolatile optical disk can be provided. In these cases, each drive can be connected to a bus (not shown) by one or more data media interfaces.

[0018] The communication unit 140 enables communication with another computing device via a communication medium. Additionally, the functions of the components of the computing device 100 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 100 can operate in a networked environment using a logical connection with one or more other servers, a personal computer (PC), or another general network node.

[0019] Input device 150 may be one or more various input devices, such as a mouse, keyboard, tracking ball, voice input device, etc. Output device 160 may be one or more output devices, such as a display, a speaker, a printer, etc. Computing device 100 may also communicate with one or more external devices (not shown) through communication unit 140 as needed, such as storage devices, display devices, etc., communicate with one or more devices that allow a user to interact with computing device 100, or communicate with any device that allows computing device 100 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0020] In some implementations, in addition to being integrated on a single device, some or all of the various components of the computing device 100 can also be set in the form of a cloud computing architecture. In a cloud computing architecture, these components can be remotely arranged and can work together to implement the functions described in the present disclosure. In some implementations, cloud computing provides computing, software, data access and storage services, which do not require end users to know the physical location or configuration of the system or hardware that provides these services. In various implementations, cloud computing uses appropriate protocols to provide services through a wide area network (such as the Internet). For example, a cloud computing provider provides applications through a wide area network, and they can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data can be stored on a server at a remote location. The computing resources in a cloud computing environment can be merged at a remote data center location or they can be dispersed. Cloud computing infrastructure can provide services through a shared data center, even if they appear as a single access point for users. Therefore, the components and functions described herein can be provided from a service provider at a remote location using a cloud computing architecture. Alternatively, they can be provided from a conventional server, or they can be installed directly or otherwise on a client device.

[0021] The computing device 100 may be used to implement a scheme for a convolutional neural network (CNN) according to multiple implementations of the present disclosure. The computing device 100 may receive input data, such as an image, etc., through an input device 150. The CNN module 122 may process the input data to obtain corresponding output data, such as a classification of the image. The output data may be provided to an output device 160 to be provided as an output 180 to a user, etc. For example, the classification of the image (e.g., dog, cat, etc.) may be displayed on a display to be presented to the user.

[0022] Exploiting sparsity in CNN is a key technique to reduce the cost of model inference. Many CNNs have a large number of zero or near-zero values ​​in their calculations. By skipping these zero values, the model inference cost or overhead can be significantly reduced. In neural network inference, there is usually sparsity in three aspects: weights, inputs, and outputs. Here we focus on the sparsity prediction of outputs.

[0023] According to multiple implementations of the present disclosure, a highly quantized version (e.g., less than 4 bits) of the original CNN network is first run to predict a sparse mask (e.g., a binary sparse mask) of the output feature map, which indicates the location of non-zero items in the output feature map. The binary sparse mask can then be used to guide full-precision convolution, for example, to calculate only the values ​​of the corresponding non-zero items without calculating the zero-valued items. Since the original network is quantized, the method does not require pre-training or external knowledge.

[0024] In some implementations, quantized prediction can accurately predict output sparsity with very little computational overhead. In addition, full-precision sparse convolution can effectively exploit this sparsity to accelerate inference, for example, by adapting the hardware platform for sparse convolution.

[0025] Figure 2 A schematic diagram of an architecture 200 of a convolutional neural network according to some implementations of the present disclosure is shown. In the architecture 200, an input image (e.g., an image of a cat) is converted into an input feature map 202. The input feature map 202 is passed through a convolution layer 204 to generate a feature map 206 as the output feature map of the layer. The feature map 206 is provided to the next convolution layer 208 to generate a feature map 210 as the output feature map of the layer. The feature map 210 is provided to a soft maximization layer (Softmax) layer 212 to output the classification result of the convolutional neural network. Figure 2 As shown, "cat" has the highest probability, in other words, the input image is classified as "cat". The application of convolutional neural networks is described above in conjunction with classification applications. However, it should be understood that implementations according to the present disclosure may also be applied to any other suitable type of application.

[0026] The sparse prediction module 220 illustrates a sparse implementation of the convolution layer 204 according to some implementations of the present disclosure. It should be understood that the sparse prediction module 220 can also be applied to any other suitable convolution layer. In this implementation, the input feature map (F) 222 can be the input feature map 202, which can be provided to the sparse convolution layer (S-Conv) 224 with the weights (W). The sparse convolution layer 224 also receives a sparse mask (M) 252 from the prediction module 240, which indicates the sparsity of the output feature map 230 of the convolution layer. In other words, if the sparse mask 252 indicates that the corresponding output in the output feature map (F') 230 is zero, there is no need to calculate the value of the corresponding output. The output of the sparse convolution layer 224 is provided to an activation function, such as a linear rectifier function (ReLU) layer 226 or a maximum pooling (Max-pooling) layer 228. The output of the activation function is the output feature map 230, which corresponds to the feature map 206 in this implementation.

[0027] The prediction module 240 illustrates a method for determining a sparse mask according to some implementations of the present disclosure, which is a quantized version of the convolutional layer 204. The input feature map 222 is quantized into a quantized input feature map (Fq) 242. The quantized input feature map 242 is provided to a quantized version of the convolutional neural network. For example, the quantized input feature map 242 is provided to a quantized convolutional layer (Q-Conv) 244. The quantized input feature map 242 is convolved with the quantized weights Wq to obtain a quantized output feature map (F'q) 246. The quantized output feature map 246 is provided to a quantized activation function, such as a quantized ReLu (Q-ReLU) layer 248 or a quantized maximum pooling (Q-max-pooling) layer 250. The output of the activation function is a sparse mask 252 for providing to the sparse convolutional layer 224 in the module 220.

[0028] Figure 3 300 according to some implementations of the present disclosure. In the quantization operation 300, the maximum absolute value M in the feature map (or tensor) 302 may be first determined, and then the value x may be calculated by the formula is mapped to x', where n represents the number of bits of the quantization operation, and In this way, a quantized version of the feature map 302, ie, the feature map 304, can be obtained.

[0029] exist Figure 3 In the example, the feature map 302 may be as follows: Figure 2 The input feature map 222 shown in FIG. 1 and the feature map 304 may be as shown in FIG. Figure 2Quantized input feature map 242 is shown. In this example, a 3x3 tensor is quantized to 4 bits. It should be understood that any other quantization with more or less bits may also be used. In some implementations, no more than 4 bits of quantization (e.g., 4 bits or 1 bit) may be used, which is significantly lower than the number of bits of conventional quantization operations (usually above 8 bits).

[0030] exist Figure 3 In , we can first determine the maximum absolute value of the tensor, which in this example is "1.2". Then, we can define a linear mapping function to map the maximum value to the largest integer representation, that is, 2 4-1 -1 = 7. Therefore, any number between -1.2 and 1.2 is linearly mapped to -8 to +7 by the mapping function as described above.

[0031] The above describes the quantization operation in conjunction with a specific quantization method, however, it should be understood that any other suitable quantization method may also be used. Quantization is a common method for accelerating the inference and training of neural networks. Unlike traditional quantization, quantization is not used here to perform complete inference, but quantization is used to predict the sparsity of the output feature map layer by layer. Therefore, compared with the quantization scheme that implements complete classification, a quantization operation with much lower bits can be used.

[0032] For the ReLU activation function, the prediction module 240 can determine the sign of each element in the output feature map and return those negative values ​​to zero. For Max-pooling, the prediction module 240 can determine the location of the maximum value in each sub-region without considering its exact value. Lower quantization bits generally mean lower computational overhead and faster inference speed. However, over-quantization can also introduce too much prediction error and reduce classification accuracy. The optimal quantization level can be determined empirically, or the lowest quantization level can be determined experimentally. For example, it was found through implementation that top-1 accuracy can be achieved with 4-bit quantization.

[0033] The following is a detailed description of Figure 2 The quantized convolution layer 224 is also called a quantized convolution operation Q-Conv. Unlike conventional methods, the quantized convolution operation Q-Conv does not require a deconvolution stage to restore accuracy, and therefore requires fewer operations and runs faster.

[0034] Equation (1) represents the traditional convolution operation:

[0035]

[0036] in Denotes a convolution operation, W denotes the weight of the convolution operation, X denotes the input data of the convolution operation, Y denotes the output data of the convolution operation, and N denotes the dimension of the data. For simplicity, only the weight is considered here, and the bias is omitted. Given a quantization function f, the quantized convolution operation is shown by equation (2).

[0037]

[0038] in represents an integer convolution operation, f represents a quantization function, and f -1 Represents the inverse function of the quantization function.

[0039] Unlike traditional quantized convolution, the Q-Conv operation no longer requires dequantization. This is because: for ReLU, the prediction module 240 only considers the sign of each element, while for Max-pooling, the prediction module 240 only considers the position of the maximum value of each sub-region.

[0040] Therefore, the calculation formula is shown in equation (3). It can be seen that the Q-Conv operation no longer requires dequantization

[0041]

[0042] In many common CNN models, convolutional layers are usually followed by batch normalization (BN) layers and / or ReLU layers and / or Max-pooling layers. Different models have different combinations of these layers, depending on how they are designed and tuned. For sparse mask-based predictions, all these combinations can be divided into two groups.

[0043] 1) Convolution + ReLU and / or Max-pooling

[0044] As mentioned above, Q-Conv outputs low-bit integers. When the ReLU layer is connected after the convolution, a quantized ReLU operation (Q-ReLU) can be applied on the output of Q-Conv. Q-ReLU only cares about the sign of the output feature map of Q-Conv, and generates a corresponding sparse mask with the same dimension. Similarly, Q-Max-pooling only cares about the position of the maximum value in the sub-region and generates a corresponding sparse mask.

[0045] Figure 4 4 is a schematic diagram of a method 400 for determining a sparse mask according to some implementations of the present disclosure. For example, the method 400 can be used to implement Figure 2 The quantized convolution layer 224 is shown. Figure 4As shown, the output feature map 402 of Q-Conv is converted into a feature map 404 through a Q-ReLU operation, and then converted into a sparse mask 406. Similarly, the output feature map 402 of Q-Conv is converted into a feature map 408 through a Q-Max-pooling operation (2x2), and then converted into a sparse mask 410.

[0046] 2) Convolution + Batch Normalization + ReLU / Max-pooling

[0047] In some implementations, batch normalization is very common in CNN models and can be applied to reduce the covariance shift within the feature map. Unlike ReLU and Max-pooling layers, the batch normalization layer has four parameters, including the scaling factor α, bias β, average mean μ, and average variance σ 2 , as shown in equation (4).

[0048] As shown in equation (5), two parameters need to be quantized, the convolution weight W i and batch normalization scaling factor α j Applying quantized batch normalization (Q-BN) directly on the output of Q-Conv will result in the multiplication of two quantized parameters. This will amplify the quantization error and produce additional errors in sparsity prediction.

[0049] According to the implementation of the present disclosure, quantization error can be removed by fusing Q-Conv kernel and Q-BN kernel. Kernel fusion is a common method for accelerating deep neural network (DNN) models. Here, convolution and batch normalization are fused to remove compound quantization error. Equation (6) shows the derivation of the fusion operation, where α is replaced by j and W i Fusion is f(α j ×W i ), and the fused Q-Conv and Q-BN operation is called Q-Conv-BN.

[0050]

[0051]

[0052]

[0053] As shown in equation (6), by performing quantization after kernel fusion, compound quantization errors can be avoided. In this way, loss of accuracy can be avoided while reducing the computational overhead of the prediction module 240.

[0054] The above introduces how to use the sparsity of feature maps to save computational overhead in combination with convolutional neural networks. Theoretically, given a ReLU layer with 80% sparsity, the upper limit of acceleration is 5 times (saving 80% of the computation). For the Max-pooling layer, 2x2 Max-pooling can save 3 / 4 of the computation, which means a 4-fold acceleration. However, in actual operation, due to the quantization prediction cost and sparse computational overhead, how to convert the sparsity of feature maps into acceleration on real hardware platforms still needs further consideration.

[0055] According to some implementations of the present disclosure, quantized sparse prediction and sparse convolution methods can be accelerated on commercial hardware or general hardware. Current commercial central processing units (CPUs) generally do not have native low-bit arithmetic hardware support. Therefore, the vector processing unit of the CPU (e.g., the advanced vector extension instruction set AVX) can be used to perform quantized prediction. For example, some CPUs do not have native support for 4-bit data, and 8-bit integers can be used for arithmetic operations, even if a lower bit precision (e.g., 4 bits) is used in the prediction network. In an implementation using 4-bit quantization, a 4-bit format can be used to cache intermediate results for efficient storage.

[0056] Figure 5 FIG. 5 is a schematic diagram of an encoding method 500 according to some implementations of the present disclosure. Figure 5 As shown, the sparse mask 502 includes three layers, each layer corresponding to a corresponding layer or slice of the output feature map. The sparse mask 502 can be vectorized into a vector 504, where each row of the vector 504 represents a corresponding slice of the sparse mask 502. The vector 504 can be encoded by recording the position of the non-zero items in the vector 504. For example, the column index 506 represents the column index of each non-zero item, and the row index 508 represents the row index of the first non-zero item in each row of non-zero items.

[0057] The sparse coding format according to the implementation of the present disclosure can increase the computational efficiency of sparse convolution. Specifically, the sparse convolution (S-Conv) kernel can discard the index of all zero outputs and only take the index of non-zero entries. In addition, the matrix index can be directly encoded, so that S-Conv can retrieve the index and input vector with almost negligible overhead.

[0058] The two models, the prediction model and the sparse convolution model, cause repeated storage of quantization parameters and feature maps. The additional storage overhead leads to a waste of memory access, thereby reducing performance. According to some implementations of the present disclosure, multi-level data multiplexing technology can be used to solve this problem. In order to increase data reuse, input quantization can be fused with the execution of Q-Conv, Q-BN, Q-ReLU, and Q-Max-pooling to fully reuse data in CPU registers. In some implementations, S-Conv can also be fused with S-BN, S-ReLU, and S-Max-pooling to reuse temporary data in registers and / or CPU caches.

[0059] In some implementations, since Q-Conv is packed in AVX vector format, in-register arithmetic shifts can be used to convert data formats between AVX and other kernels.

[0060] According to some implementations of the present disclosure, specially designed hardware can be used to fully exploit the advantages of quantization and sparse convolution. For example, the mixed precision hardware can be a graphics processing unit (GPU) and / or a field programmable gate array (FPGA). For example, in a GPU, mixed precision tensor cores can further reduce quantization prediction overhead, and massively parallel processing units can accelerate sparse convolution operations.

[0061] FPGA's reconfigurable lookup table (LUT) can provide circuit-level bit manipulation, which is very suitable for low-bit quantization prediction and achieves extremely high performance. FPGA's digital signal processor (DSP) unit can implement very fast sparse convolution. The high flexibility of FPGA leaves more room for multi-level optimization, such as fine-grained load balancing and data multiplexing.

[0062] Figure 6 A flowchart of a method 600 according to some implementations of the present disclosure is shown. The method 600 may be Figure 1 The computing device 100 shown or Figure 2 The method 600 is implemented in the architecture 200 shown. Alternatively, the method 600 may also be implemented in any suitable computing environment or architecture. The method 600 may be implemented by a general-purpose processing unit or dedicated hardware. The dedicated hardware may support mixed-precision arithmetic operations and may include a massively parallel processing unit, such as a GPU or FPGA.

[0063] At 602, the input feature map is quantized to obtain a quantized input feature map. For example, the input feature map may be quantized with a bit number not exceeding 4 bits. For example, in the case where method 600 is implemented by a general-purpose processor, the quantization may be performed by a vector processing unit of the general-purpose processor. In some implementations, data format conversion between the vector processing unit and other cores of the general-purpose processor is implemented by a shift operation inside a register.

[0064] At 604, a sparse mask, such as a binary sparse mask, for an output feature map is determined by a quantized version of the convolutional neural network based on the quantized input feature map. The sparse mask may indicate the locations of non-zero entries in the output feature map. This may be accomplished by, for example, Figure 2 The prediction module 240 shown is implemented as shown. For example, a binary sparse mask is encoded to record only the indices of non-zero entries.

[0065] In some implementations, a quantized output feature map is determined by a quantized convolution operation based on the quantized input feature map, and a sparse mask is determined based on the quantized output feature map.

[0066] In some implementations, based on the quantized input feature map, a quantized output feature map is determined by a quantized convolution and batch normalization operation, wherein the convolution and batch normalization operation is a fusion of the convolution operation and the batch normalization operation. A sparse mask is determined based on the quantized output feature map.

[0067] In some implementations, the quantized output feature map is provided to an activation function to determine the output of the activation function. The sparse mask is determined based on the sign of the output of the activation function. The activation function can be ReLU or Max-pooling.

[0068] At 606, based on the input feature map, non-zero items in the output feature map indicated by the sparse mask are determined by the convolutional neural network. For example, based on the input feature map and the positions of the non-zero items indicated by the sparse mask, the values ​​of the non-zero items in the output feature map can be calculated or otherwise determined by the convolutional neural network.

[0069] In some implementations, the method 600 is implemented by a general-purpose processor, and the quantization is performed by a vector processing unit of the general-purpose processor.

[0070] In some implementations, data format conversion between the vector processing unit and other cores of the general-purpose processor is implemented through shift operations within registers.

[0071] Some example implementations of the present disclosure are listed below.

[0072] In a first aspect, the present disclosure provides a computer-implemented method. The method includes: quantizing an input feature map to obtain a quantized input feature map; determining a sparse mask for an output feature map based on the quantized input feature map by a quantized version of a convolutional neural network, the sparse mask indicating the location of non-zero items in the output feature map; and determining the non-zero items in the output feature map indicated by the sparse mask by the convolutional neural network based on the input feature map.

[0073] In some implementations, quantizing the input feature map includes quantizing the input feature map with a bit number not exceeding 4 bits.

[0074] In some implementations, determining the sparse mask includes: determining a quantized output feature map through a quantized convolution operation based on the quantized input feature map; and determining the sparse mask based on the quantized output feature map.

[0075] In some implementations, determining the sparse mask includes: determining a quantized output feature map based on the quantized input feature map through quantized convolution and batch normalization operations, wherein the convolution and batch normalization operations are a fusion of convolution operations and batch normalization operations; and determining the sparse mask based on the quantized output feature map.

[0076] In some implementations, determining the sparse mask based on the quantized output feature map includes: providing the quantized output feature map to an activation function to determine an output of the activation function; and determining the sparse mask based on a sign of the output of the activation function.

[0077] In some implementations, the sparse mask is encoded to record only the indices of non-zero entries.

[0078] In some implementations, the method is implemented by a general-purpose processor, and the quantization is performed by a vector processing unit of the general-purpose processor.

[0079] In some implementations, data format conversion between the vector processing unit and other cores of the general processor is implemented by shift operations inside registers.

[0080] In some implementations, the method is implemented by hardware supporting mixed-precision arithmetic operations.

[0081] In a second aspect, the present disclosure provides a device. The device includes: a processing unit; and a memory, coupled to the processing unit and containing instructions stored thereon, the instructions causing the device to perform the following actions when executed by the processing unit: quantizing an input feature map to obtain a quantized input feature map; based on the quantized input feature map, determining a sparse mask for an output feature map by a quantized version of a convolutional neural network, the sparse mask indicating zero-valued items and non-zero items in the output feature map; and based on the input feature map, determining the non-zero items in the output feature map indicated by the sparse mask by the convolutional neural network.

[0082] In some implementations, quantizing the input feature map includes quantizing the input feature map with a bit number not exceeding 4 bits.

[0083] In some implementations, determining the sparse mask includes: determining a quantized output feature map through a quantized convolution operation based on the quantized input feature map; and determining the sparse mask based on the quantized output feature map.

[0084] In some implementations, determining the sparse mask includes: determining a quantized output feature map based on the quantized input feature map through quantized convolution and batch normalization operations, wherein the convolution and batch normalization operations are a fusion of convolution operations and batch normalization operations; and determining the sparse mask based on the quantized output feature map.

[0085] In some implementations, determining the sparse mask based on the quantized output feature map includes: providing the quantized output feature map to an activation function to determine an output of the activation function; and determining the sparse mask based on a sign of the output of the activation function.

[0086] In some implementations, the sparse mask is encoded to record only the indices of non-zero entries.

[0087] In some implementations, the actions are implemented by a general purpose processor, and the quantization is performed by a vector processing unit of the general purpose processor.

[0088] In some implementations, data format conversion between the vector processing unit and other cores of the general processor is implemented by shift operations inside registers.

[0089] In some implementations, the actions are performed by hardware supporting mixed-precision arithmetic operations.

[0090] In a third aspect, the present disclosure provides a computer program product, which is tangibly stored in a non-transitory computer storage medium and includes computer executable instructions, which when executed by a device cause the device to perform the method in the first aspect of the present disclosure.

[0091] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a device, cause the device to perform the method in the first aspect of the present disclosure.

[0092] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0093] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0094] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0095] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate implementation can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0096] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A computer-implemented method comprising: quantizing the input feature map to obtain a quantized input feature map; Based on the quantized input feature map, determining a sparse mask for an output feature map by a quantized version of a convolutional neural network, the sparse mask indicating locations of non-zero entries in the output feature map; as well as Based on the input feature map, non-zero entries in the output feature map indicated by the sparse mask are determined by the convolutional neural network.

2. The method according to claim 1, wherein quantizing the input feature map comprises: The input feature map is quantized with a bit number not exceeding 4 bits.

3. The method of claim 1 , wherein determining the sparse mask comprises: Based on the quantized input feature map, determining a quantized output feature map through a quantized convolution operation; as well as The sparse mask is determined based on the quantized output feature map.

4. The method of claim 1 , wherein determining the sparse mask comprises: Based on the quantized input feature map, determining a quantized output feature map through quantized convolution and batch normalization operations, wherein the convolution and batch normalization operations are a fusion of convolution operations and batch normalization operations; as well as The sparse mask is determined based on the quantized output feature map.

5. The method according to claim 3 or 4, wherein determining the sparse mask based on the quantized output feature map comprises: Providing the quantized output feature map to an activation function to determine an output of the activation function; as well as The sparse mask is determined based on a sign of an output of the activation function. The method according to claim 1 , wherein the sparse mask is encoded to record only the indices of non-zero entries.

7. The method according to claim 1, wherein the method is implemented by a general-purpose processor, and the quantization is performed by a vector processing unit of the general-purpose processor. 8 . The method according to claim 7 , wherein the data format conversion between the vector processing unit and other cores of the general processor is implemented by a shift operation inside a register.

9. The method of claim 1, wherein the method is implemented by hardware supporting mixed-precision arithmetic operations.

10. A device comprising: Processing unit; as well as A memory coupled to the processing unit and containing instructions stored thereon, which, when executed by the processing unit, cause the device to perform the following actions: quantizing the input feature map to obtain a quantized input feature map; Based on the quantized input feature map, determining a sparse mask for an output feature map by a quantized version of a convolutional neural network, the sparse mask indicating locations of non-zero entries in the output feature map; as well as Based on the input feature map, non-zero entries in the output feature map indicated by the sparse mask are determined by the convolutional neural network.

11. The apparatus of claim 10, wherein quantizing the input feature map comprises: The input feature map is quantized with a bit number not exceeding 4 bits.

12. The apparatus of claim 10, wherein determining the sparse mask comprises: Based on the quantized input feature map, determining a quantized output feature map through a quantized convolution operation; as well as The sparse mask is determined based on the quantized output feature map.

13. The apparatus of claim 10, wherein determining the sparse mask comprises: Based on the quantized input feature map, determining a quantized output feature map through quantized convolution and batch normalization operations, wherein the convolution and batch normalization operations are a fusion of convolution operations and batch normalization operations; as well as The sparse mask is determined based on the quantized output feature map.

14. The apparatus of claim 12 or 13, wherein determining the sparse mask based on the quantized output feature map comprises: Providing the quantized output feature map to an activation function to determine an output of the activation function; as well as The sparse mask is determined based on a sign of an output of the activation function.

15. The apparatus of claim 10, wherein the sparse mask is encoded to record only indices of non-zero entries.

16. The apparatus of claim 10, wherein the actions are implemented by a general purpose processor and the quantization is performed by a vector processing unit of the general purpose processor.

17. The device according to claim 16, wherein the data format conversion between the vector processing unit and other cores of the general processor is implemented by a shift operation inside a register.

18. The apparatus of claim 10, wherein the actions are implemented by hardware supporting mixed-precision arithmetic operations.

19. A computer program product stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform actions comprising: quantizing the input feature map to obtain a quantized input feature map; Based on the quantized input feature map, determining a sparse mask for an output feature map by a quantized version of a convolutional neural network, the sparse mask indicating locations of non-zero entries in the output feature map; as well as Based on the input feature map, non-zero entries in the output feature map indicated by the sparse mask are determined by the convolutional neural network.

20. The computer program product of claim 19, wherein quantizing the input feature map comprises: The input feature map is quantized with a bit number not exceeding 4 bits.

Images