Fractional bit quantization and deployment of convolutional neural network models

Abstract

The invention relates to fractional bit network quantization and deployment of CNN models. An AI accelerator includes: an input buffer configured to buffer an input image; a weight buffer configured to buffer a convolution kernel index for a convolutional layer of the CNN model; a kernel mode buffer configured to buffer a 1-bit convolution kernel subset for a convolutional layer of the CNN model, where the 1-bit convolution kernel subset includes 2 [tau] 1-bit convolution kernels of a size of K * K; a PE array comprising one or more PE nodes, each PE node configured to generate a convolution result of an image region of the input image and a one-bit convolution kernel corresponding to the convolution kernel index in the subset of one-bit convolution kernels; and an output buffer configured to buffer a convolution result of each image region of the input image and the one-bit convolution kernel corresponding to the convolution kernel index.

Description

Technical Field

[0001] Embodiments described herein relate generally to the field of neural networks, and more particularly to fractional-bit network quantization and deployment of convolutional neural network (CNN) models. Background Art

[0002] Convolutional neural network (CNN) models are powerful learning models that have achieved state-of-the-art performance for many computer vision tasks. CNN models include an input layer, an output layer, and at least one hidden layer in between, and use complex mathematical models to process data transmitted between these network layers. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Various advantages of the embodiments will become apparent to those skilled in the art upon reading the following description and appended claims and referring to the following drawings, in which:

[0004] Figure 1A-1B is a flowchart illustrating a method for quantizing a CNN model according to some embodiments of the present disclosure;

[0005] Figure 2 is a diagram showing the acceleration of a fractional-bit CNN model relative to a 1-bit CNN model according to some embodiments of the present disclosure;

[0006] Figure 3 is a block diagram illustrating a fractional artificial intelligence (AI) accelerator for deploying a fractional CNN model according to some embodiments of the present disclosure;

[0007] Figure 4 is a diagram showing some embodiments of the present disclosure Figure 3 A block diagram of a processing engine (PE) node in a fractional-bit AI accelerator is shown;

[0008] Figure 5 is a block diagram illustrating components according to some example embodiments of the present disclosure, which components are capable of reading instructions from a machine-readable or computer-readable medium and performing any one or more of the methods discussed herein;

[0009] Figure 6 is a block diagram of an example processor platform according to some embodiments of the present disclosure; and

[0010] Fig. 7A is a diagram showing the various layers within a CNN model.

[0011] Figure 7B is a diagram showing exemplary computational stages within a convolutional layer of a CNN model. DETAILED DESCRIPTION

[0012] The various aspects of the illustrative embodiments will be described using the terms commonly used by those skilled in the art, so as to convey the substance of the present disclosure to other persons skilled in the art. However, it will be apparent to those skilled in the art that many alternative embodiments may be implemented using the parts of the described aspects. For the purpose of explanation, specific numbers, materials, and configurations are set forth to provide a comprehensive understanding of the illustrative embodiments. However, it will be apparent to those skilled in the art that alternative embodiments may be implemented without these specific details. In other cases, known features may be omitted or simplified to avoid blurring the illustrative embodiments.

[0013] In addition, various operations are described as multiple discrete operations in sequence in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be interpreted as implying that these operations are necessarily order-dependent. Specifically, these operations do not need to be performed in the order presented.

[0014] A CNN model is a specialized feed-forward neural network model designed to process data with a known grid-like topology, such as image data. As a result, CNN models are often used in computational vision and image recognition applications, but they can also be used for other types of pattern recognition, such as speech and language processing.

[0015] Fig. 7A is a diagram showing the various layers within a CNN model. Fig. 7A As shown, an exemplary CNN model for modeling image processing can receive an input 702 describing the red, green, and blue (RGB) components of an input image. The input 702 can be processed by multiple convolutional layers (e.g., a first convolutional layer 704, a second convolutional layer 706). The outputs from the multiple convolutional layers can be optionally processed by a set of fully connected layers 708. The neurons in the fully connected layer have full connections for all activations in the previous layer, as previously described for the feedforward network. The outputs from the fully connected layer 708 can be used to generate the output results of the network. Matrix multiplication can be used instead of convolution to calculate the activations within the fully connected layer 708. Not all CNN implementations use the fully connected layer 708. For example, in some embodiments, the second convolutional layer 706 can generate the output of the CNN model.

[0016] The convolutional layers are sparsely connected, which is different from the traditional neural network configuration established in the fully connected layer 708. Traditional neural network layers are fully connected so that each output unit interacts with every input unit. However, the convolutional layers are sparsely connected because the output of the convolution of a field (rather than the corresponding state value of each node in the field) is input to the nodes of the subsequent layer, as shown in the figure. The convolution kernels associated with the convolutional layers perform convolution operations, and the outputs of these convolution operations are sent to the next layer. The dimensionality reduction performed within the convolutional layers is one aspect that enables CNN models to scale to process large images.

[0017] Figure 7B is a diagram showing exemplary computational stages within a convolutional layer of a CNN model. An input 712 to a convolutional layer of a CNN model may be processed in three stages of a convolutional layer 714. The three stages may include a convolutional stage 716, a detector phase 718, and a pooling stage 720. The convolutional layer 714 may then output data to a subsequent convolutional layer. The last convolutional layer of the network may generate output feature map data or provide input to a fully connected layer, for example, to generate a classification value for the input to the CNN model.

[0018] Several convolutions are performed in parallel in the convolution stage 716 to produce a set of linear activations. The convolution stage 716 may include an affine transformation, which is any transformation that can be specified as a linear transformation plus a translation. Affine transformations include rotations, translations, scaling, and combinations of these transformations. The convolution stage calculates the output of a function (e.g., a neuron) connected to a specific region in the input, which can be determined as a local region associated with the neuron. The neuron calculates the dot product between the weight of the neuron and the region in the local input to which the neuron is connected. The output from the convolution stage 716 defines a set of linear activations that are processed by subsequent stages of the convolution layer 714.

[0019] The linear activations may be processed by the detector stage 718. In the detector stage 718, each linear activation is processed by a nonlinear activation function. The nonlinear activation function increases the nonlinearity of the entire network without affecting the receptive field of the convolutional layer. Several types of nonlinear activation functions may be used. One specific type is the rectified linear unit (ReLU), which uses an activation function defined as f(x) = max(0, x) such that the activation threshold is zero.

[0020] The pooling stage 720 uses a pooling function that replaces the output of the second convolutional layer 706 with summary statistics of nearby outputs. The pooling function can be used to introduce translation invariance into the neural network so that a small translation of the input does not change the pooled output. In the case where the presence of features in the input data is more important than the precise location of the features, invariance to local translation can be useful. Various types of pooling functions can be used during the pooling stage 720, including maximum pooling, average pooling, and l2 norm pooling. In addition, some CNN implementations do not include a pooling stage. Instead, this implementation replaces an additional convolution stage (which has an increased stride | relative to the previous convolution stage).

[0021] The output from the convolutional layer 714 may then be processed by the next layer 722. The next layer 722 may be an additional convolutional layer or a fully connected layer 708. For example, Fig. 7A The first convolution layer 704 can output to the second convolution layer 706, and the second convolution layer can output to the first layer in the fully connected layer 708.

[0022] CNN models have achieved record-breaking results in almost all computer vision tasks. The leading performance of CNN models is attributed to the huge number of learnable parameters (even up to hundreds of millions of learnable parameters), regardless of the availability of large-scale datasets and powerful hardware resources. However, this in turn brings heavy costs in storage, computation, and power, which limit their widespread use, especially on resource-constrained devices. In order to enable CNN models to be applied to mobile, embedded or other low-power devices, both academia and industry have conducted a lot of research on network quantization of CNN models.

[0023] The extreme quantization solution for CNN models is 1-bit network quantization, in which the parameters of the CNN model are forced to be 1-bit values ​​{-1,1}. Specifically, the basic concept of 1-bit network quantization is as follows: Assume M = {f(W l ,A l ;X l )|1≤l≤L} is a 32-bit floating-point CNN model (it should be understood that the 32-bit floating-point CNN model is only an example, and it can also be a 64-bit or 16-bit floating-point CNN model or a block floating-point (BFP)-16CNN model), where W l is the 32-bit floating-point weight set of the lth convolutional layer, X l is the input set of the lth convolutional layer, A lis the activation set of the lth convolutional layer, and L is the number of convolutional layers in the 32-bit floating-point CNN model M. 1-bit network quantization aims to convert the 32-bit floating-point CNN model M into a 1-bit CNN model with only the quantized weight set Its entries are represented by α l B, where B∈{1,-1}, α l is a positive scaling factor specific to the lth convolutional layer. Mathematically, the objective function for 1-bit network quantization of a 32-bit floating-point CNN model M can be defined as follows:

[0024]

[0025] The optimization of the objective function (1) can be easily solved using, for example, the popular straight-through estimator (STE).

[0026] Compared to 32-bit floating-point CNN models, 1-bit network quantization significantly reduces storage and latency (32x compression and up to 58x acceleration) by replacing the original time-intensive multiplication operations with cheap bitwise XNOR and bitcount operations, simplifying the deployment of CNN solutions, especially for dedicated AI devices.

[0027] Existing 1-bit network quantization solutions (including but not limited to Binary-Weight-Network (BWN), DoReFa-Net, XNOR-Net, Ternary-Binary Network (TBN), Loss-aware Binary Network (LBN), Explicit Loss-error-aware Quantization (ELQ), Learned Quantization (LQ)-Net, Differentiable Soft Quantization (DSQ) and Information Retention (IR)-Net) focus on binary quantization scheme design, aiming to obtain high-precision 1-bit CNN models.

[0028] Benefiting from the great progress in binary network design in recent years, modern binary networks have a small accuracy degradation relative to the full-precision reference model. For single-class or multi-class vision applications such as face detection / verification, pedestrian / human detection, etc., 1-bit CNN models have been widely used because the accuracy degradation is negligible (usually less than 1%). However, there is still considerable room for compression and acceleration of 1-bit CNN models without losing model accuracy.

[0029] Figure 1A is a flowchart showing a method 100A for quantifying a CNN model according to some embodiments of the present disclosure. As Figure 1A shown, the method 100A includes, for the l-th convolutional layer (l is a positive integer) of the CNN model: S102A, allocating a 1-bit convolutional kernel subset P l to the l-th convolutional layer, where the l-th convolutional layer includes a 32-bit or 64-bit floating-point convolutional kernel of size K×K, and the 1-bit convolutional kernel subset P l includes 2 τ 1-bit convolutional kernels of size K×K, 1 ≤ τ < K×K, and both K and τ are positive integers; and S104A, performing weight quantization of the l-th convolutional layer by selecting a 1-bit convolutional kernel from the 1-bit convolutional kernel subset P l as the 1-bit convolutional kernel of the l-th convolutional layer.

[0030] In some embodiments, the 1-bit convolutional kernel subset P l can be shared by all convolutional layers of the CNN model or specific to the l-th convolutional layer. The 1-bit convolutional kernel subsets allocated to different convolutional layers of the CNN model can include the same number or different numbers of 1-bit convolutional kernels. The 1-bit convolutional kernels of the 1-bit convolutional kernel subset P l can be predefined or randomly selected from a 1-bit convolutional kernel set U l including all possible 1-bit convolutional kernels of size K×K or a part thereof.

[0031] In some embodiments, the objective function for network quantization of the CNN model can be defined as follows:

[0032]

[0033] where, is the 1-bit convolutional kernel subset allocated to the l-th convolutional layer, is the set of quantized weights of the l-th convolutional layer, and W l is the set of 32-bit or 64-bit floating-point weights of the l-th convolutional layer.

[0034] During the process of training the CNN model, the 1-bit convolutional kernel subset P l can be improved by replacing one or more 1-bit convolutional kernels in it with one or more remaining 1-bit convolutional kernels in the 1-bit convolutional kernel set U l . l .

[0035] Figure 1B is a flowchart showing a method 100B for quantifying a CNN model according to some embodiments of the present disclosure. As Figure 1BAs shown, the method 100B includes, for the lth convolution layer of the CNN model: S102B, based on the 1-bit convolution kernel set U l and the selection matrix S l Determine the 1-bit convolution kernel subset P for the lth convolution layer l ; and S104B, by selecting from a 1-bit convolution kernel subset P l Select a 1-bit convolution kernel as the 1-bit convolution kernel of the l-th convolution layer and perform weight quantization of the l-th convolution layer.

[0036] In some embodiments, the 1-bit convolution kernel subset P l Determined as follows:

[0037] P l =U l S l (3)

[0038] in, To determine the 1-bit convolution kernel subset for the l-th convolutional layer, is a set of 1-bit convolution kernels including all possible 1-bit convolution kernels of size K×K, To select the matrix, it is randomly initialized with parameter τ and is fixed for the lth convolutional layer.

[0039] It should be understood that as defined in Equation 3, the 1-bit convolution kernel set U l Select a 1-bit convolution kernel subset P from l Depends on the 1-bit convolution kernel set U l The permutation of the 1-bit convolution kernel in , and the 1-bit convolution kernel set U needs to be permuted according to the predefined permutation rule l The 1-bit convolution kernels in are permuted to select the required 1-bit convolution kernel subset P l .

[0040] In some embodiments, the 1-bit convolution kernel subset P l Determined as follows:

[0041] P l =U l RS l (4)

[0042] in, To determine the subset of 1-bit convolution kernels used for the l-th convolutional layer, is a set of 1-bit convolution kernels including all possible 1-bit convolution kernels of size K×K, is the selection matrix, R is the permutation matrix set R l The permutation matrix selected in .

[0043] In some embodiments, the objective function of the network quantization of the CNN model is defined as follows:

[0044]

[0045] Among them, and are a set of permutation matrices, is the set of quantized weights of the l-th convolutional layer, W l is the set of 32-bit or 64-bit floating-point weights of the l-th convolutional layer.

[0046] Compared with the 1-bit CNN model, the CNN model generated from the quantization of Method 100A / 100B is compressed by approximately K×K / τ times. Therefore, the quantization of Method 100A / 100B is called fractional-bit network quantization, and the CNN model generated from the quantization of Method 100A / 100B is called a fractional-bit CNN model. Specifically, in the 1-bit CNN model, a 1-bit convolutional kernel of size K×K requires K×K bits to represent all its weights, and according to the layout of {1, -1} values, there are a total of 2 K×K candidates for the 1-bit convolutional kernel. In contrast, in the fractional-bit CNN model, the number of candidates for the 1-bit convolutional kernel of the 1-bit convolutional kernel is forced to be 2 τ , 1 ≤ τ < K×K. On average, it only takes τ bits to represent all the weights of a 1-bit convolutional kernel of size K×K, rather than K×K bits.

[0047] A large number of experiments show that compared with the 1-bit CNN model, the model accuracy of the fractional-bit CNN model is not lost. In addition, compared with Method 100A, Method 100B is a more advanced quantization solution, which formulates the selection formula of the subset P l of the 1-bit convolutional kernel as a new permutation learning problem solved in an end-to-end manner, generating a fractional-bit CNN model with higher accuracy.

[0048] Figure 2 is an illustration showing the acceleration of the fractional-bit CNN model according to some embodiments of the present disclosure relative to the 1-bit CNN model. For example, assume that C out is the number of output feature channels of the l-th convolutional layer, and C in is the number of input feature channels of the l-th convolutional layer. In the 1-bit CNN model, the K×K kernel-wise 1-bit convolution operation for each pixel is performed C out ×C in times. In contrast, in the fractional-bit CNN model, the K×K kernel-wise 1-bit convolution operation for each pixel is performed 2 τ ×C in times because 2 τ 1-bit convolutional kernels are shared by all convolutional kernels, each convolutional kernel corresponding to one output feature channel, and 2 τ << Cout 。

[0049] Based on the concept of fractional-bit network quantization, a fractional-bit AI accelerator is proposed for deploying the fractional-bit CNN models generated from Method 100A / 100B. Specifically, the fractional-bit AI accelerator seamlessly bridges the characteristics of the fractional-bit CNN models into the hardware (HW) accelerator architecture design, and innovates in the following aspects: (1) the connection of processing engines (PEs) and the design of distributed buffers, (2) the design of pre-computation and accumulation working mechanisms, and (3) the design of compact dual-buffer lookup tables (LUTs) to achieve an efficient computing process.

[0050] Figure 3 is a block diagram showing a fractional-bit AI accelerator 300 for deploying fractional-bit CNN models according to some embodiments of the present disclosure. As Figure 3 shown, the fractional-bit AI accelerator 300 includes an input buffer 302, a weight buffer 304, a kernel mode buffer 306, a PE array 308, and an output buffer 310, where: the input buffer 302 is configured to buffer input images (i.e., each input feature channel of the input image); the weight buffer 304 is configured to buffer convolution kernel indices for the convolutional layers of the fractional-bit CNN model; the kernel mode buffer 306 is configured to buffer a 1-bit convolutional kernel subset for the convolutional layers of the fractional-bit CNN model, where the 1-bit convolutional kernel subset includes 2 τ 1-bit convolutional kernels of size K×K, 1 ≤ τ < K×K, and where both K and τ are positive integers; the PE array 308 includes one or more PE nodes, each PE node being configured to generate a convolution result of an image region of the input image (i.e., input values in each input feature channel of the image region) and a 1-bit convolutional kernel corresponding to the convolution kernel index in the 1-bit convolutional kernel subset, and output the generated convolution result to the output buffer 310; the output buffer 310 is configured to buffer the convolution results of each image region of the input image and the 1-bit convolutional kernels corresponding to the convolution kernel indices.

[0051] For example, when the fractional-bit CNN model generated from Method 100A / 100B is deployed in the fractional-bit AI accelerator 300, the weight buffer 304 is configured to buffer the convolution kernel indices for the l-th convolutional layer, and these convolution kernel indices are broadcast to all PE nodes in the PE array 308; the kernel mode buffer 306 is configured to buffer a 1-bit convolutional kernel subset P τ including 2 l 1-bit convolutional kernels, and these 1-bit convolutional kernels are broadcast to all PE nodes in the PE array 308; and each PE node in the PE array 308 is configured to generate a certain image region of the input image and the 1-bit convolutional kernel subset P lThe convolution result of the 1-bit convolution kernel corresponding to the convolution kernel index used for the l-th convolution layer.

[0052] In other words, in the fractional AI accelerator 300, the reasoning of the fractional CNN model includes the following steps: 1) loading the input values ​​of each image region of the input image and the weights of a certain convolutional layer from the corresponding buffer to the PE node of the PE array; 2) assigning the computing tasks associated with each image region of the input image to the PE nodes in the PE array that serve as parallel computing units; 3) outputting the computing results associated with each image region of the input image to the output buffer.

[0053] In the fractional-bit AI accelerator 300 , since the convolution task associated with the input image is split into multiple subtasks associated with various image regions of the input image that can be performed in parallel, the utilization rate of the hardware resources of the fractional-bit AI accelerator 300 can be maximized.

[0054] Figure 4 is a diagram showing some embodiments of the present disclosure Figure 3 FIG. 4 is a block diagram of a PE node 400 in a fractional-bit AI accelerator. Figure 4 As shown, the PE node 400 includes an activation loader 402, a local input buffer 404, a weight loader 406, a local weight buffer 408, a dot product unit 410, a lookup table (LUT) unit 412, at least two accumulators 414, a local output buffer 416, and an output writer 418.

[0055] Specifically, when the fractional-bit CNN model generated from method 100A / 100B is deployed in the fractional-bit AI accelerator 300: the activation loader 402 is configured to load the input value associated with a certain image region of the input image from the input buffer 302 into the local input buffer 404. The weight loader 406 is configured to load the convolution kernel index for the lth convolution layer into the local weight buffer 408, wherein the convolution kernel index associated with different output feature channels is divided into P groups, each group includes the convolution kernel index associated with T output feature channels, and these convolution kernel indexes are buffered in a separate bank of the local weight buffer 408. The dot product unit 410 is configured to obtain the 1-bit convolution kernel subset P from the kernel pattern buffer 306. l , obtain the input value associated with the image region of the input image from the local input buffer 404, generate the input value associated with the image region of the input image and the 1-bit convolution kernel subset P l 2 of 2 τThe at least two accumulators 414 are configured to accumulate dot products associated with the same output feature channel and output the accumulated results associated with a corresponding set of output feature channels to a corresponding bank of the local output buffer 416. The output writer 418 is configured to read the accumulated results associated with different output feature channels from the accumulator 414 and write them to the output buffer 310.

[0056] The architecture of the fractional-bit AI accelerator 300 can be implemented on a field programmable gate array (FPGA) because it can efficiently utilize a large amount of on-chip distributed memory (Block RAM). Assuming that we need to calculate and create a 3×3 input LUT on each input feature channel, both the hardware architecture design and the scheduling strategy need to use input reservation, that is, maximize the life of the input data of each input feature channel within the PE node. For 3×3 input, all related output feature channels are calculated before switching to the 3×3 input of the next input feature channel. On the other hand, there is a mapping relationship between the PE node and the input feature channel, so that the LUT associated with each input feature channel of the corresponding image area can be saved in the same PE node without sharing with other PE nodes, thereby reducing the complexity of the design.

[0057] like Figure 4 As shown, PE node 400 includes two pipeline designs: pre-calculation and accumulation. Pre-calculation is implemented by dot product unit 410 based on two data inputs, one data input comes from kernel pattern buffer 305 shared by all PE nodes, and the other data input comes from local input buffer 410. Before starting the calculation, activation loader 402 loads the input value associated with the image region of the input image into local input buffer 404. The global addressing logic shared by all PE nodes traverses each entry in kernel pattern buffer 306 and adds 2 of the 1-bit convolution kernel subset to the kernel pattern buffer 306. τ The 1-bit convolution kernel is sent to the dot product unit 410.

[0058] In the PE node 400, a double-buffer LUT with one write port and multiple read ports can be used as the LUT unit 412. In each cycle, the dot product unit 410 writes the current dot product to the write buffer (WBuffer) of the LUT unit 412, and the corresponding accumulator reads the previous dot product from the read buffer (RBuffer) of the LUT unit 412 in sequence. τ After cycles, with 2 of the 1-bit convolution kernel subset τ The dot products associated with the 1-bit convolution kernels are written to the corresponding accumulator, which produces an accumulated result associated with the image region.

[0059] Figure 5 is a block diagram illustrating components according to some example embodiments that are capable of reading instructions from a machine-readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium) and performing any one or more of the methods discussed herein. Specifically, Figure 5 A schematic diagram of hardware resources 500 is shown, which includes one or more processors (or processor cores) 510, one or more memory / storage devices 520, and one or more communication resources 530, each of which may be communicatively coupled via a bus 540. For embodiments utilizing node virtualization (e.g., NFV), a hypervisor 502 may be executed to provide an execution environment for one or more network slices / sub-slices to utilize the hardware resources 500.

[0060] Processor 510 may include, for example, processor 512 and processor 514, which may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a vision processing unit (VPU), a field programmable gate array (FPGA), or any suitable combination thereof.

[0061] The memory / storage device 520 may include a main memory, a disk storage device, or any suitable combination thereof. The memory / storage device 520 may include, but is not limited to, any type of volatile or non-volatile memory, such as, for example, a dynamic random access memory (DRAM), a static random access memory (SRAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory, etc.

[0062] The communication resources 530 may include an interconnect, a network interface component, or other suitable device to communicate with one or more peripheral devices 504 or one or more databases 506 via the network 508. For example, the communication resources 530 may include a wired communication component (e.g., for coupling via a universal serial bus (USB)), a cellular communication component, an NFC component, Components (e.g. Low energy consumption), components, and other communication components.

[0063] The instructions 550 may include software, programs, applications, applet, applications, or other executable code for causing at least any one of the processors 510 to perform any one or more of the methods discussed herein. The instructions 550 may reside in whole or in part in at least one of the processors 510 (e.g., in a cache of the processor), the memory / storage device 520, or any suitable combination thereof. In addition, any portion of the instructions 550 may be transferred to the hardware resources 500 from any combination of the peripheral devices 504 or the database 506. Therefore, the memory of the processor 510, the memory / storage device 520, the peripheral devices 504, and the database 506 are examples of computer-readable and machine-readable media.

[0064] Figure 6 600 is a block diagram of an example processor platform according to some embodiments of the present disclosure. The processor platform 600 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, a tablet computer (e.g., an iPad TM )), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, headphones or other wearable device, or any other type of computing device.

[0065] The processor platform 600 of the illustrated example includes a processor 612. The processor 612 of the illustrated example is hardware. For example, the processor 612 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers of any desired series or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In some embodiments, the processor implements one or more of the above methods or processes.

[0066] The processor 612 of the illustrated example includes a local memory 613 (e.g., cache). The processor 612 of the illustrated example communicates with a main memory including a volatile memory 614 and a non-volatile memory 616 via a bus 618. The volatile memory 614 may be comprised of a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 616 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 614, 616 is controlled by a memory controller.

[0067] The processor platform 600 of the illustrated example also includes an interface circuit 620. The interface circuit 620 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), interface, a near field communication (NFC) interface, and / or a PCI Express interface.

[0068] In the example shown, one or more input devices 622 are connected to the interface circuit 620. The input device(s) 622 allow a user to enter data and / or commands into the processor 612. The input device(s) may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, and / or a voice recognition system.

[0069] One or more output devices 624 are also connected to the interface circuit 620 of the illustrated example. The output device 624 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit 620 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0070] The interface circuitry 620 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces, to facilitate the exchange of data with external machines (e.g., any type of computing device) over a network 626. Communications may be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, etc.

[0071] For example, the interface circuit 620 may include a training data set input via one or more input devices 622 or retrieved from the network 626.

[0072] The processor platform 600 of the illustrated example also includes one or more mass storage devices 628 for storing software and / or data. Examples of such mass storage devices 628 include floppy disk drives, hard disk drives, optical disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0073] Machine-executable instructions 632 may be stored in the mass storage device 628, volatile memory 614, non-volatile memory 616, and / or removable non-transitory computer-readable storage media (e.g., CD or DVD).

[0074] Additional notes and examples:

[0075] Example 1 includes an artificial intelligence (AI) accelerator, comprising: an input buffer configured to buffer an input image; a weight buffer configured to buffer convolution kernel indices for a convolutional layer of a convolutional neural network (CNN) model; a kernel pattern buffer configured to buffer a 1-bit convolutional kernel subset for a convolutional layer of the CNN model, where the 1-bit convolutional kernel subset includes 2 τ 1-bit convolutional kernels of size K×K, 1 ≤ τ < K×K, and where K and τ are positive integers; a processing engine (PE) array including one or more PE nodes, each PE node being configured to generate a convolution result of an image region of the input image and a 1-bit convolutional kernel corresponding to the convolution kernel index in the 1-bit convolutional kernel subset; and an output buffer configured to buffer convolution results of respective image regions of the input image and 1-bit convolutional kernels corresponding to the convolution kernel indices.

[0076] Example 2 includes the AI accelerator of Example 1, wherein different PE nodes generate convolution results of different image regions of the input image and 1-bit convolutional kernels corresponding to the convolution kernel indices, and the same PE nodes generate convolution results of the same image regions of the input image and 1-bit convolutional kernels corresponding to the convolution kernel indices.

[0077] Example 3 includes the AI accelerator of Example 2, wherein each PE node in one or more PE nodes includes: a dot product unit configured to generate a dot product of an image region of the input image and 2 τ 1-bit convolutional kernels; a look-up table (LUT) unit configured to buffer an image region of the input image and 2 τThe method comprises the steps of: performing a dot product of a 1-bit convolution kernel and finding the dot product associated with the 1-bit convolution kernel corresponding to the convolution kernel index; and at least two accumulators, each accumulator being configured to accumulate the dot products associated with the same output feature channel among the dot products associated with the 1-bit convolution kernel corresponding to the convolution kernel index.

[0078] Example 4 includes the AI ​​accelerator of Example 3, wherein the LUT unit is a double-buffer LUT having one write port and at least two read ports.

[0079] Example 5 includes the AI ​​accelerator of Example 1, wherein the 1-bit convolution kernel subset is shared by all convolution layers of the CNN model.

[0080] Example 6 includes the AI ​​accelerator of Example 1, wherein the 1-bit convolution kernel subset is specific to the convolution layer of the CNN model.

[0081] Example 7 includes the AI ​​accelerator of Example 6, wherein the 1-bit convolution kernel subsets specific to different convolution layers of the CNN model include the same number of 1-bit convolution kernels.

[0082] Example 8 includes the AI ​​accelerator of Example 6, wherein the 1-bit convolution kernel subsets specific to different convolution layers of the CNN model include different numbers of 1-bit convolution kernels.

[0083] Example 9 includes the AI ​​accelerator of Example 1, wherein 2 of the 1-bit convolution kernel subset τ The 1-bit convolution kernels are randomly selected from a set of 1-bit convolution kernels including all possible 1-bit convolution kernels of size K×K or a portion thereof.

[0084] Example 10 includes the AI ​​accelerator of Example 1, wherein an objective function of network quantization associated with the CNN model is defined as follows:

[0085]

[0086] in, is the 1-bit convolution kernel subset used in the convolutional layer of the CNN model, is the quantized weight set of the convolutional layer of the CNN model, W l A set of 32-bit or 64-bit floating-point weights of a convolutional layer of another CNN model quantized to a CNN model.

[0087] Example 11 includes the AI ​​accelerator of Example 1, wherein a subset of 1-bit convolution kernels for a convolution layer of a CNN model is determined as follows:

[0088] P l =U l S l

[0089] in, is the 1-bit convolution kernel subset used in the convolutional layer of the CNN model, is a set of 1-bit convolution kernels including all possible 1-bit convolution kernels of size K×K, is the selection matrix.

[0090] Example 12 includes the AI ​​accelerator of Example 1, wherein a subset of 1-bit convolution kernels for a convolution layer of a CNN model is determined as follows:

[0091] P l =U l RS l

[0092] in, is the 1-bit convolution kernel subset used in the convolutional layer of the CNN model, is a set of 1-bit convolution kernels including all possible 1-bit convolution kernels of size K×K, is the selection matrix of the convolutional layer specific to the CNN model, and R is the permutation matrix selected from the permutation matrix set.

[0093] Example 13 includes the AI ​​accelerator of Example 12, wherein an objective function of network quantization associated with the CNN model is defined as follows:

[0094]

[0095] in, and is a set of permutation matrices, is the quantized weight set of the convolutional layer of the CNN model, W l A set of 32-bit or 64-bit floating-point weights of a convolutional layer of another CNN model quantized to a CNN model.

[0096] Example 14 includes a method for quantizing a convolutional neural network (CNN) model, comprising performing the following operations for a convolutional layer of the CNN model: determining a subset of 1-bit convolution kernels for the convolutional layer based on a 1-bit convolution kernel set and a selection matrix, wherein the convolutional layer includes 32-bit or 64-bit floating-point convolution kernels of size K×K, and the 1-bit convolution kernel set includes all possible 1-bit convolution kernels of size K×K, where K is a positive integer; and performing weight quantization of the convolutional layer by selecting a 1-bit convolution kernel from the 1-bit convolution kernel set as the 1-bit convolution kernel of the convolutional layer.

[0097] Example 15 includes the method of Example 14, wherein the 1-bit convolution kernel subset is determined as follows:

[0098] P l =U l S l

[0099] in, is a 1-bit convolution kernel subset, is a 1-bit convolution kernel set, is the selection matrix.

[0100] Example 16 includes the method of Example 14, wherein the 1-bit convolution kernel subset is determined as follows:

[0101] P l =U l RS l

[0102] in, is a 1-bit convolution kernel subset, is a 1-bit convolution kernel set, is the selection matrix, and R is the permutation matrix selected from the permutation matrix set.

[0103] Example 17 includes the method of Example 16, wherein the objective function for network quantization of the CNN model is defined as follows:

[0104]

[0105] in, and is a set of permutation matrices, is the quantized weight set of the convolutional layer, W l is the set of 32-bit or 64-bit floating-point weights for the convolutional layer.

[0106] Example 18 includes a computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor circuit, cause the processor circuit to perform the method of any of Examples 14-17.

[0107] Example 19 includes a device for a convolutional neural network (CNN), including a device for performing the method described in any of Examples 14 to 17.

[0108] Example 20 includes a device for a convolutional neural network (CNN), comprising: a memory having instructions stored thereon; and a processor circuit coupled to the memory, wherein the instructions, when executed by the processor circuit, cause the processor circuit to perform any of the methods of Examples 14 to 17.

[0109] The above detailed description includes reference to the accompanying drawings, which form a part of the detailed description. The accompanying drawings show specific embodiments that can be implemented by way of example. These embodiments are also referred to as "examples" in this article. These examples may include elements other than the elements shown or described. However, the inventors also contemplate that only examples of the elements shown or described are provided. In addition, the inventors also contemplate embodiments using any combination or permutation of those elements (or one or more aspects thereof) shown or described, whether with respect to a specific embodiment (or one or more aspects thereof), or with respect to other embodiments (or one or more aspects thereof) shown or described herein.

[0110] All publications, patents, and patent documents mentioned in this document are incorporated herein by reference in their entirety, as if individually incorporated by reference. If there is an inconsistency in usage between this document and a document incorporated by reference, the usage in the introduced (one or more) references should be considered to supplement this document; for irreconcilable inconsistencies, the usage in this document shall prevail.

[0111] In this document, as is common in patent documents, the terms "a" or "an" are used to include one or more than one, independent of any other instance or use of "at least one" or "one or more". In this document, unless otherwise stated, the term "or" is used to refer to a non-exclusive or, such that "A or B" includes "A but not B", "B but not A", and "A and B". In the appended claims, the terms "including" and "wherein" are used as the plain English equivalents of the respective terms "comprising" and "wherein". In addition, in the following claims, the terms "including" and "comprising" are open-ended, that is, a system, device, article, or process that includes elements other than those listed after such terms in a claim is still deemed to fall within the scope of the claim. In addition, in the following claims, the terms "first", "second", "third", etc. are used merely as labels and are not intended to impose numerical requirements on their objects.

[0112] The above description is intended to be illustrative, not restrictive. For example, the above examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments may be used, such as those used by those of ordinary skill in the art after reading the above description. The abstract is intended to enable the reader to quickly understand the nature of the present technology disclosure, and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above specific embodiments, various features may be combined together to simplify the disclosure. This should not be interpreted as unclaimed public features being necessary for any claim. On the contrary, the subject matter of the invention may be present in less than all the features of the specific disclosed embodiments. Therefore, the following claims are incorporated into the specific embodiments, wherein each claim is independent of a separate embodiment. The scope of the embodiment should be determined with reference to the appended claims and the full scope of equivalents to which these claims are assigned.

Claims

1. An artificial intelligence (AI) accelerator, comprising: an input buffer configured to buffer an input image; A weight buffer configured to buffer convolution kernel indices for a convolution layer of a convolutional neural network (CNN) model; A kernel mode buffer configured to buffer a subset of 1-bit convolution kernels for a convolutional layer of the CNN model, where the subset of 1-bit convolution kernels includes two 1-bit convolution kernels of size K×K, 1≤τ<K×K, and where K and τ are positive integers; τ wherein, K and τ are positive integers; a processing engine (PE) array, comprising one or more PE nodes, each PE node being configured to generate a convolution result of an image region of the input image and a 1-bit convolution kernel corresponding to the convolution kernel index in the 1-bit convolution kernel subset; and An output buffer is configured to buffer the convolution results of each image region of the input image with the 1-bit convolution kernel corresponding to the convolution kernel index.

2. The AI ​​accelerator according to claim 1, wherein: Different PE nodes generate convolution results of different image areas of the input image and the 1-bit convolution kernel corresponding to the convolution kernel index, and the same PE node generates convolution results of the same image area of ​​the input image and the 1-bit convolution kernel corresponding to the convolution kernel index.

3. The AI ​​accelerator according to claim 2, wherein: Each of the one or more PE nodes includes: A dot product unit is configured to generate an image region of the input image and the 2 τ The dot product of a 1-bit convolution kernel; A lookup table (LUT) unit configured to buffer the image region of the input image and the 2 τ dot products of a 1-bit convolution kernel and finding the dot product associated with the 1-bit convolution kernel corresponding to the convolution kernel index; and At least two accumulators, each accumulator configured to accumulate dot products associated with the same output feature channel among the dot products associated with the 1-bit convolution kernel corresponding to the convolution kernel index.

4. The AI ​​accelerator according to claim 3, wherein: The LUT unit is a double buffer LUT having one write port and at least two read ports.

5. The AI ​​accelerator according to claim 1, wherein: The 1-bit convolution kernel subset is shared by all convolutional layers of the CNN model.

6. The AI ​​accelerator according to claim 1, wherein: The 1-bit convolution kernel subset is specific to the convolutional layers of the CNN model.

7. The AI ​​accelerator according to claim 6, wherein: The 1-bit convolution kernel subsets specific to different convolutional layers of the CNN model include the same number of 1-bit convolution kernels.

8. The AI ​​accelerator according to claim 6, wherein: The 1-bit convolution kernel subsets specific to different convolutional layers of the CNN model include different numbers of 1-bit convolution kernels.

9. The AI ​​accelerator according to claim 1, wherein: The 2 of the 1-bit convolution kernel subset τ The 1-bit convolution kernels are randomly selected from a set of 1-bit convolution kernels including all possible 1-bit convolution kernels of size K×K or a portion thereof.

10. The AI ​​accelerator according to claim 1, wherein: The objective function of network quantization associated with the CNN model is defined as follows: in, is the 1-bit convolution kernel subset used for the convolutional layer of the CNN model, is the quantized weight set of the convolutional layer of the CNN model, W l A set of 32-bit or 64-bit floating point weights of a convolutional layer of another CNN model quantized to this CNN model.

11. The AI ​​accelerator according to claim 1, wherein: The 1-bit convolution kernel subset for the convolutional layer of the CNN model is determined as follows: P l =U l S l in, is the 1-bit convolution kernel subset used for the convolutional layer of the CNN model, is a set of 1-bit convolution kernels including all possible 1-bit convolution kernels of size K×K, is the selection matrix.

12. The AI ​​accelerator according to claim 1, wherein: The 1-bit convolution kernel subset for the convolutional layer of the CNN model is determined as follows: P l =U l RS l in, is the 1-bit convolution kernel subset used for the convolutional layer of the CNN model, is a set of 1-bit convolution kernels including all possible 1-bit convolution kernels of size K×K, is a selection matrix specific to the convolutional layer of the CNN model, and R is a permutation matrix selected from a set of permutation matrices.

13. The AI ​​accelerator according to claim 12, wherein: The objective function of network quantization associated with the CNN model is defined as follows: in, and is the permutation matrix set, is the quantized weight set of the convolutional layer of the CNN model, W l A set of 32-bit or 64-bit floating point weights of a convolutional layer of another CNN model quantized to this CNN model.

14. A method for quantizing a convolutional neural network (CNN) model, comprising performing the following operations on a convolutional layer of the CNN model: A 1-bit convolution kernel subset for the convolution layer is determined based on the 1-bit convolution kernel set and the selection matrix, wherein: The convolution layer includes a 32-bit or 64-bit floating-point convolution kernel of size K×K, the 1-bit convolution kernel set includes all possible 1-bit convolution kernels of size K×K, K is a positive integer; and By selecting a 1-bit convolution kernel from the 1-bit convolution kernel set as the 1-bit convolution kernel of the convolution layer, weight quantization of the convolution layer is performed.

15. The method according to claim 14, wherein: The 1-bit convolution kernel subset is determined as follows: P l =U l S l in, is the 1-bit convolution kernel subset, is the 1-bit convolution kernel set, is the selection matrix.

16. The method according to claim 14, wherein: The 1-bit convolution kernel subset is determined as follows: P l =U l RS l in, is the 1-bit convolution kernel subset, is the 1-bit convolution kernel set, is the selection matrix, and R is a permutation matrix selected from a permutation matrix set.

17. The method according to claim 16, wherein: The objective function for network quantization of the CNN model is defined as follows: in, and is the permutation matrix set, is the quantized weight set of the convolutional layer, W l is the set of 32-bit or 64-bit floating-point weights of the convolutional layer.

18. A computer readable medium having instructions stored thereon, wherein: The instructions, when executed by a processor circuit, cause the processor circuit to perform the method of any one of claims 14 to 17.

19. An apparatus for a convolutional neural network (CNN), comprising means for performing the method of any one of claims 14 to 17.

20. An apparatus for a convolutional neural network (CNN), comprising: a memory having instructions stored therein; as well as A processor circuit coupled to the memory, wherein the instructions, when executed by the processor circuit, cause the processor circuit to perform the method of any one of claims 14 to 17.

Images