A sparse convolution accelerator based on hybrid bit-width

Abstract

The application discloses a sparse convolution accelerator based on a mixed bit width, which comprises a total control module, a DMA, a first calculation core and a cache module.The total control module is used for controlling the overall operation of the sparse convolution accelerator based on the mixed bit width; the DMA is used for carrying out the carrying of image data and weights in and out of a chip; the first calculation core is used for completing the acceleration of the sparse convolution layer of the mixed bit width; and the cache module is used for caching and multiplexing data.The application proposes a CNN multiplication accumulator structure which takes a low-bit-width multiplier as a basic unit, supports mixed precision calculation through addition, shifting and flexible combination of the basic unit, and flexibly and efficiently realizes the hardware acceleration of convolution operators of different compression network models.The application can be widely applied to the field of chip design.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of chip design, and in particular to a sparse convolution accelerator based on mixed bit width. BACKGROUND

[0002] As the most popular deep learning model at present, the convolutional neural network (CNN) has been widely applied in the AI field such as image processing, target detection, audio recognition, etc.

[0003] Convolution is the basic operator of convolutional neural network, and is realized by a multiply-accumulate unit on hardware. The traditional CNN multiply-accumulate unit is used for processing fixed format data multiplication and accumulation, such as int16 and int8 multiplication. However, when implementing int4 and int2 multiplication, using int16 and int8 multiplier will waste a large number of multiplier bit widths, and the multiplier cannot be used efficiently. In addition, with the development of neural network model compression technology, it has become a trend to deploy convolutional neural network using sparse data format with mixed precision. The data format of convolution operation may be mixed with int8, int4 and int2 types, and there are a large number of data or bits with zero value.

[0004] Therefore, there is an urgent need for a flexible sparse convolution multiply-accumulate unit supporting mixed precision. SUMMARY

[0005] Therefore, in order to solve the defects of low flexibility of the traditional convolutional neural network multiply-accumulate unit and difficulty in efficiently performing convolution operation of mixed data format, the present application provides a sparse convolution accelerator based on mixed bit width, comprising:

[0006] A general control module is configured to control the overall operation of the sparse convolution accelerator based on mixed bit width.

[0007] A direct memory access (DMA) is configured to perform image data and weight transfer within and outside the chip.

[0008] A first calculation core is configured to complete the acceleration of the sparse convolution layer with mixed bit width.

[0009] A cache module is configured to cache and reuse data.

[0010] In some embodiments, the sparse convolution accelerator based on mixed bit width further comprises:

[0011] A second calculation core is configured to complete the calculation of the activation function and the normalization layer.

[0012] In some embodiments, the first computing core is provided with a plurality of processing element units (PEs), which include signed multipliers, left shift 0 / 2 multiplexers, left shift 0 / 4 multiplexers, and a configurable shift register.

[0013] Through the preferred embodiment, a convolution calculation engine is proposed, which performs bit-wise summation and re-shift accumulation on serially input weight bit units.

[0014] In some embodiments, in the processing element unit, 32 signed multipliers are included, wherein:

[0015] The 32 signed multipliers are divided into 4 groups;

[0016] When the input activation and the weight are 8 bits, the processing element unit performs convolution multiplication and accumulation of 8 input channels in parallel each time.

[0017] In some embodiments, when the input activation and the weight are 8 bits, the processing element unit performs convolution multiplication and accumulation of 8 input channels in parallel each time, which specifically includes:

[0018] In the first period, the processing element unit inputs 8-bit input activation of 8 channels and the lowest bit unit of the corresponding 8-channel weight, and 8 signed multipliers in each group receive 8-channel data;

[0019] In the second period, the input activation of the processing element unit is kept unchanged, and the next-lowest bit unit of the corresponding 8-channel weight is input to the processing element unit;

[0020] In the third period, the input activation of the processing element unit is kept unchanged, and the next-higher bit unit of the corresponding 8-channel weight is input to the processing element unit;

[0021] In the fourth period, the input activation of the processing element unit is kept unchanged, and the highest bit unit of the corresponding 8-channel weight is input to the processing element unit, and the multiplication and accumulation calculation of 8 channels is completed.

[0022] According to the weight serial period, the configurable shift register is left shifted by 0, 2, 4, and 6 respectively in four periods.

[0023] In the embodiment, to expand the support of mixed 2, 4, and 8-bit input activation and weight data formats, through flexible data mapping of the 4 groups of signed multipliers, a total of 12 kinds of data format activation and weight multiplication and accumulation calculation can be completed.

[0024] In some embodiments, a multiplication and accumulation array is constructed based on the processing element unit, wherein:

[0025] Input activation is quantized by layer, and weight data is quantized by input channel group.

[0026] In some embodiments, further comprising:

[0027] A weight flag is added to process asynchronous control data flow.

[0028] For weight data, weight FIFO is input to PE, and the same weight data is multiplexed by each column of PE, and different columns of PE calculate different output channels of the output feature map. Due to the difference in quantization compression rate of different convolution kernels, different columns of PE will work asynchronously. Therefore, by adding a weight flag to process asynchronous control data flow, the total control module can complete the control of asynchronous data flow.

[0029] In some embodiments, further comprising:

[0030] The data to be multiplied is divided into 2-bit units.

[0031] Any 8, 4, 6, or 2-bit data can be split into multiple 2-bit basic units for shift addition.

[0032] In some embodiments, further comprising:

[0033] Sign extension is performed before signed multiplication.

[0034] In this embodiment, considering that the input activation format needs to support signed numbers or unsigned numbers, and the weight needs to support signed numbers, additional sign extension is required before signed multiplication.

[0035] Based on the above scheme, the application provides a sparse convolution accelerator based on mixed bit width. Through flexible splitting and combination of low-bit multipliers, different data formats and different sparseness of multiplication and accumulation calculation can be efficiently supported, and high hardware utilization is maintained. Further, an asynchronous convolution calculation array is combined based on the calculation engine unit, and high data multiplexing rate and time and space utilization of the calculation array are maintained through data mapping. Finally, a CNN convolution accelerator architecture design supporting mixed precision and bit-level sparsity is completed. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a working process diagram of a sparse convolution accelerator based on mixed bit width of the application;

[0037] Figure 2 is a calculation process diagram of a calculation example of an embodiment of the application;

[0038] Figure 3 is a structural block diagram of a calculation engine unit of an embodiment of the application;

[0039] Figure 4 is a schematic diagram of a data processing process of a multiply-accumulate array of a specific embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0041] It should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0042] It should be understood that the “system”, “device”, “unit” and / or “module” used in the present application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0043] Unless the context clearly indicates otherwise, the words “one”, “an”, “a” and / or “the” do not refer to the singular, but can also include the plural. Generally, the terms “comprise” and “include” only indicate that the steps and elements explicitly identified are included, and these steps and elements do not constitute an exclusive list. The method or device can also include other steps or elements. The element defined by the statement “comprising a” does not exclude the presence of additional identical elements in the process, method, product or device comprising the element.

[0044] In the description of the embodiments of the present application, “a plurality of” means two or more than two. The following terms “first”, “second” are only used for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with “first”, “second” can explicitly or implicitly include one or more of the features.

[0045] In the image processing process, the specific scene can be target detection based on a convolutional neural network, and the extracted feature map data and the convolution kernel parameters are used as inputs of the accelerator.

[0046] An optional example of the sparse convolution accelerator based on the mixed bit width proposed in the present application includes but is not limited to the following modules:

[0047] A total control module for controlling the overall operation of the mixed-bit-width-based sparse convolution accelerator;

[0048] A DMA for performing the transfer of image data and weights in and out of the chip;

[0049] A first computing core for completing the acceleration of the sparse convolution layer of mixed bit width;

[0050] A second computing core for completing the calculation of the activation function and the normalization layer;

[0051] A cache module for caching and multiplexing data.

[0052] The first computing core is provided with a plurality of computing engine units, and the computing engine units include 32 3bit*3bit signed multipliers, 2 left shift 0 / 2 multiplexers, 1 left shift 0 / 4 multiplexer and 1 multiplexer with configurable shift 0-6.

[0053] The application firstly splits arbitrary bit width data into a plurality of 2bit bit units according to the binary complement multiplication principle, only uses basic 3bit*3bit signed multipliers, and then realizes a mixed precision, bit-level sparse bit fusion multiplication and accumulation computing engine through the shift addition of the multiplication parts of the data bits at different positions and the flexible combination of the 3bit*3bit multipliers, and then constructs a multiplication and accumulation array of asynchronous convolution calculation based on the bit fusion computing engine, supports the hardware acceleration of a plurality of convolution operators through the data mapping and multiplexing mechanism, and finally completes the system integration of on-chip cache, convolution calculation array and total control, and designs a comprehensive compressed convolution neural network hardware accelerator.

[0054] In some embodiments, the computing engine unit:

[0055] The data to be multiplied is divided in 2bit units, and the sign extension is performed before the signed multiplication.

[0056] Arbitrary 8, 4, 6 and 2bit width data can be split into a plurality of 2bit basic units of shift addition, for example, 4bit width data can be split into 2 2bit units; 8bit width data can be split into 4 2bit units, based on this principle, the technology adopted by the application is called bit fusion technology, and all the data to be multiplied is divided in 2bit units.

[0057] The data to be multiplied comes from the off-chip DDR memory, including input feature maps and convolution kernel parameters, and the accelerator automatically divides and extends the sign of the data to be multiplied, and performs mixed bit width multiplication and accumulation operation.

[0058] In addition, considering that the format of the input activation needs to support signed numbers or unsigned numbers, the weight needs to support signed numbers, so additional sign extension is needed before signed multiplication. For example, -14*5=-70, the calculation process is as follows Figure 2 First, group every 2 bits of the data bit, for 8-bit number -14, it can be grouped into 4 groups, and for 4-bit number 5, it can be grouped into 2 groups. Then, for each 2-bit group, expand the sign bit to 3 bits, and the rule of expanding the sign bit is that the 2-bit group of the highest bit of the input activation and the weight is expanded to the highest bit (1 / 0), and the remaining 2-bit group is expanded to 0. Finally, using the shift addition principle of the multiplier, each 3-bit group is signed multiplied with each other, and is shifted according to the number of bits, and then is added after expanding the sign bit of the 8 partial sums, to obtain the output result.

[0059] Based on the above binary complement multiplication mathematical principle, the application provides a calculation engine unit which calculates by bit summation and then shifting accumulation, and the weight bit unit is serially inputted, and the structure of the calculation engine unit is as shown in Figure 3

[0060] In some feasible embodiments, 32 signed multipliers are divided into 4 groups, mul0-mul3:

[0061] When the input activation and the weight are 8 bits, the calculation engine unit calculates the convolution multiplication accumulation of 8 input channels in parallel each time.

[0062] Specifically, in the first period, 8-bit input activations i0-i7 of 8 channels and the lowest bit units w0[1:0]-w7[1:0] of the corresponding 8-channel weights are inputted to the PE, and 8 signed multipliers in each group simultaneously receive 8-channel data. In the second period, the input activation of the PE is kept unchanged, and the next-lowest bit units w0[3:2]-w7[3:2] of the 8-channel weights are inputted to the PE unit. In this way, in the third period, the next-high bit units w0[5:4]-w7[5:4] are inputted to the PE unit, and in the fourth period, the highest bit units w0[7:6]-w7[7:6] are inputted to the PE, and finally the multiplication accumulation calculation of 8 channels is completed.

[0063] In some feasible embodiments, the calculation engine unit further comprises:

[0064] According to the weight serial period, the multiplexer with shift 0-6 is selected to be left shifted by 0, 2, 4 and 6 respectively in the 4 periods;

[0065] ​In addition, the configurable shift 0-6 multiplexer can be reused to skip the zeros behind the least significant bits of the weight data, i.e., the configurable shift register supports bit-level sparse skip-zero of the weight data, i.e., for example, 8-bit weight 10110000 = 1011 << 4, which can be encoded as 4-bit weight value 1011 and shift number 4, so that the configurable shift register is shifted by adding the shift number, 4 and 6 in two cycles, thereby supporting convolution acceleration of bit-level sparse data.

[0066] To extend support for mixed 2, 4, and 8-bit input activation and weight data formats, a total of 12 data format activation and weight multiplication and accumulation calculations can be completed by flexible data mapping of 4 groups of signed multipliers, as shown in Table 1. The calculation channel parallelism is determined by the input activation bit width, and the calculation period is determined by the weight bit width, as shown in Table 2.

[0067] Table 1 Relationship table of input activation bit width and corresponding multiplier input bit unit

[0068] Input activation bit-width 8bit 4bit 2bit Mul0 i0[1:0]-i7[1:0] i0[1:0]-i7[1:0] i0[1:0]-i7[1:0] Mul1 i0[3:2]-i7[3:2] i0[3:2]-i7[3:2] i8[1:0]-i15[1:0] Mul2 i0[5:4]-i7[5:4] i8[1:0]-i15[1:0] i16[1:0]-i23[1:0] Mul3 i0[7:6]-i7[7:6] i8[3:2]-i15[3:2] i24[1:0]-i31[1:0]

[0069] Table 2 Relationship table of input activation bit width and related parameters

[0070] Input activation bit-width 8bit 4bit 2bit * Channel parallelism 8 16 32 * Weight bit-width 9bit 7bit 5bit 3bit Compute time 4CLK 3CLK 2CLK 1CLK

[0071] In some possible embodiments, further comprising:

[0072] The above calculation engine units are combined into a 10-row x 32-column multiplication and accumulation array.

[0073] As shown in Figure 4 , the input activation is quantized by layer, and the weight data is quantized by input channel group. A group contains 8 / 16 / 32 input channels. The input activation is mapped by a shift register, and when the convolution step S = 1, the first 10 data of the shift register are fixedly mapped to 10 rows of PEs, and each row of PEs reuses the same input activation; when the convolution step S = 2, the first 10 odd-numbered data of the shift register are fixedly mapped to 10 rows of PEs, and through the shift register structure, most of the convolution kernel sizes and convolution steps can be adapted.

[0074] For weight data, 32-column weight FIFO (First in, First out, first-in first-out memory) is input to 32-column PE, wherein 1 set of weight FIFO includes 8 / 16 / 32 channels, the same weight data is multiplexed in each column PE, different column PEs calculate different output channels of the output feature map, due to the difference in quantization compression rate of different convolution kernels, 32-column PEs will work asynchronously, in addition, each group of weights is offline coded, the common bit skip zero number and maximum significant bit width that can represent a group of weights are extracted, and an asynchronous control data stream is processed by adding a weight flag FIFO, which includes 2-bit weight bit width, which is used to represent the effective bit width of the weight group; 3-bit bit skip zero number is used to complete bit-level sparse skip zero by 0-6 configurable shift registers; 1-bit channel (accumulation) end flag indicates that a column of asynchronous convolution (multiplication and accumulation) calculation is completed, which helps the total controller to control the asynchronous data stream. Specifically, the PE selects the output of the multi-bit shifter after a group of multi-bit multiplication through the weight bit width and the bit skip zero number, and clears the accumulator through the accumulation end flag to continuously perform the multiplication and accumulation calculation of the next output feature point.

[0075] wherein the weight FIFO is input and output weight data, and the weight flag FIFO is the information coded by the weight group.

[0076] Based on the structure of the above sparse convolution accelerator, the working principle is basically as follows:

[0077] Referring to Figure 1 First, the convolutional neural network (such as VGG16, ResNet18, etc.) needs to be trained with specific quantization awareness, and the weight parameters of the compressed network model are obtained, and then the weight parameters are uploaded to the DDR3 SDRAM after sparse coding. When performing model inference, the CPU sends instructions to the total control module to control the accelerator to start working, and the total control module dispatches secondary instructions to the cache module, DMA, first calculation core and second calculation core. Among them, the DMA is responsible for completing the transfer of image data and weights in and out of the chip; the cache module is responsible for caching and multiplexing data for the calculation core to calculate, and the cache module includes input cache, output and partial cache, weight and group flag cache; the first calculation core is used to complete the acceleration of sparse convolution layers with mixed bit width, and the second calculation core is used to complete the calculation of activation functions, normalization layers and other calculations.

[0078] The bit fusion-based mixed precision convolution hardware accelerator architecture of the application can complete most of the convolution acceleration, and 12 kinds of data formats supported include input activation support: 2 / 4 / 8 bit signed or unsigned number, and weight data support: 3 / 5 / 7 / 9 bit signed number.

[0079] At present, the data formats of the edge deployment of the convolutional neural network include floating point numbers and fixed point numbers, the floating point numbers have high precision, but the floating point multiplier needs to consume a large amount of hardware resources, and is not suitable for low-power embedded devices, therefore, 16bit and 8bit quantization of the convolutional neural network becomes a necessary option for the edge deployment of the CNN, and the corresponding hardware accelerator also uses a fixed format fixed point multiplier as the basic component of the convolution calculation unit. However, with the development of the CNN model compression technology, it has become possible to use lower bit width data formats such as 4bit and 2bit under the premise of maintaining precision, and even due to the redundancy of the CNN parameters, the data presents more and more sparsity, the traditional fixed format fixed point multiplier does not take advantage of this feature, and the hardware utilization is low when processing low bit data or sparse data. The application aims at the defects of the traditional convolutional neural network multiplication accumulator based on fixed data format, such as low flexibility, difficulty in efficiently performing mixed data format convolution operation, and the problem of not fully utilizing the sparsity of the network model, and proposes a CNN multiplication accumulator structure taking a low bit width multiplier as a basic unit, supporting mixed precision calculation through addition, shifting and flexible combination of the basic unit, and the data bit level sparse zero operation can also be realized through the configurable shift register. Further, the bit fusion-based multiplication accumulator is composed into an asynchronous convolution calculation array, which can realize efficient edge deployment of the mixed precision sparse convolutional neural network, so as to fully utilize the characteristics of the network model compression technology, and flexibly and efficiently realize the hardware acceleration of the convolution operator of different compressed network models.

[0080] The above is a specific description of the preferred embodiment of the application, but the application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the application.

Claims

1. A hybrid bit-width based sparse convolution accelerator, comprising: Comprise: A total control module for controlling the overall operation of the mixed bit width based sparse convolution accelerator; A DMA for performing the transfer of image data and weights inside and outside the chip; A first calculation core for completing the acceleration of the mixed bit width based sparse convolution layer; A cache module for caching and multiplexing data; A second calculation core for completing the calculation of the activation function and the normalization layer; A plurality of calculation engine units are arranged in the first calculation core, and the calculation engine units comprise a signed multiplier, a left shift 0 / 2 multiplexer, a left shift 0 / 4 multiplexer, and a configurable shift register; In the calculation engine unit, 32 signed multipliers are included, wherein: The 32 signed multipliers are divided into 4 groups; When the input activation and the weight are 8 bits, the calculation engine unit calculates the convolution multiplication accumulation of 8 input channels in parallel each time; When the input activation and the weight are 8 bits, the calculation engine unit calculates the convolution multiplication accumulation of 8 input channels in parallel each time, which specifically comprises: In the first period, the calculation engine unit inputs 8-bit input activation of 8 channels and the lowest bit unit of the corresponding 8-channel weight, and 8 signed multipliers in each group receive 8-channel data; In the second period, the input activation of the calculation engine unit remains unchanged, and the next-lowest bit unit of the corresponding 8-channel weight is input to the calculation engine unit; In the third period, the input activation of the calculation engine unit remains unchanged, and the next-higher bit unit of the corresponding 8-channel weight is input to the calculation engine unit; In the fourth period, the input activation of the calculation engine unit remains unchanged, and the highest bit unit of the corresponding 8-channel weight is input to the calculation engine unit, and the multiplication accumulation calculation of 8 channels is completed; According to the weight serial period, the configurable shift register is left shifted by 0, 2, 4, and 6 respectively in four periods; Based on the calculation engine unit, a multiplication accumulation array is constructed, wherein: The input activation is quantized by layer, and the weight data is quantized by input channel group.

2. The sparse convolution accelerator based on hybrid bit-widths of claim 1, wherein, Further comprise: Increase the weight flag processing asynchronous control data stream.

3. The sparse convolution accelerator based on hybrid bit-widths according to claim 2, wherein, Further comprise.

4. The data to be multiplied is divided into 2-bit units.

5. The sparse convolution accelerator based on hybrid bit-widths according to claim 3, wherein, Further comprise: Sign extension is performed before signed multiplication. Further comprise: In the first period, the calculation engine unit inputs 8-bit input activation of 8 channels and the lowest bit unit of the corresponding 8-channel weight, and 8 signed multipliers in each group receive 8-channel data; In the second period, the input activation of the calculation engine unit remains unchanged, and the next-lowest bit unit of the corresponding 8-channel weight is input to the calculation engine unit; In the third period, the input activation of the calculation engine unit remains unchanged, and the next-higher bit unit of the corresponding 8-channel weight is input to the calculation engine unit; In the fourth period, the input activation of the calculation engine unit remains unchanged, and the highest bit unit of the corresponding 8-channel weight is input to the calculation engine unit, and the multiplication accumulation calculation of 8 channels is completed; According to the weight serial period, the configurable shift register is left shifted by 0, 2, 4, and 6 respectively in four periods; Based on the calculation engine unit, a multiplication accumulation array is constructed, wherein: The input activation is quantized by layer, and the weight data is quantized by input channel group.

Images