Sparse compression method and device for inter-layer feature map of convolutional neural network

Abstract

The invention discloses a sparse compression method and device for an interlayer feature map of a convolutional neural network. The method comprises the following steps of: performing exponential bit increment compression on non-zero elements of a convolutional inter-layer feature map, recording column index information of the non-zero elements and recording a line-feed zero mark element at the same time, retaining the zero mark element if the previous element of the zero mark element is smaller than the latter element, and otherwise, removing the zero mark element. The compression device is as follows: a sparse matrix row compression module is used for extracting non-zero elements and recording corresponding column indexes at the same time, and recording a zero mark element during data row feed; and the redundant zero-labeled element filtering module is used for reserving the zero-labeled element when the column index before the zero-labeled element in the column index is smaller than the column index after the zero-labeled element. And the non-zero element increment compression module is used for carrying out statistics on the obtained index bit median and compressing the index bits of the floating point data so as to reduce the data bit width. Meanwhile, whether the non-zero elements are subjected to incremental compression or not is judged, and an incremental compression mark code is generated.

Description

Technical Field

[0001] The present invention relates to the field of convolutional neural network accelerators based on field programmable gate arrays (FPGAs), and specifically to compressing inter-layer feature maps of convolutional neural networks by using a compression method based on inter-layer feature map data column indexes combined with statistical incremental compression of single-precision floating-point exponent bits, so as to reduce data movement between FPGAs and off-chip memories. Background Art

[0002] Convolutional Neural Network (CNN) is a branch of deep learning that can learn from a large number of images to extract advanced features. In recent years, with the development of deep learning, CNN has made significant progress in the computer field and has been widely used in tasks such as image classification, object detection, and image segmentation. However, CNN has the characteristics of high computational complexity and large number of parameters, which makes it difficult to deploy CNN on edge devices.

[0003] For edge devices, the main problem of deploying CNN is the memory access of data, which will seriously affect the throughput and energy efficiency of the device. Researchers have proposed a series of solutions to this problem, which can be roughly divided into two categories. One is network optimization, which is to optimize the neural network itself. Among them, network pruning technology can prune the initial weights of the network, remove unnecessary weights and retain weights that have an impact on the training task, thereby reducing the computational workload of the network. However, although network pruning reduces the computational workload, the uneven pruning may increase the complexity of deploying CNN. Since the neural network itself is trained with 32-bit floating point numbers (FP), some optimizations also convert the 32-bit floating point numbers of network parameters into fixed-point numbers or integer data to reduce the computational workload and memory access of the network. The other category is the hardware acceleration method that focuses on hardware implementation. From the perspective of the hardware accelerator itself, researchers usually use highly parallel computing and reused computing data streams to improve the throughput and energy efficiency of the accelerator. Row-smooth data flow is a classic method. This method reduces frequent data movement and reduces the energy consumption of the accelerator by highly reusing data and network weights.

[0004] Most CNNs use the Rectified Linear Unit (ReLU) as the activation function. After the convolution data passes through the ReLU layer, the negative data is judged to be zero, and the positive data remains unchanged, which makes the inter-layer data highly sparse. In addition, the zero and non-zero values ​​of the inter-layer data occupy the same bit width, so some studies use compression coding to reduce the memory storage of inter-layer data, thereby reducing the consumption of storage resources and reducing the data movement between on-chip and off-chip memory. Some previous studies have proposed methods such as zero-value compression and incremental compression, which compress inter-layer data by saving the number of zeros between non-zero elements of the feature map and the central tendency of statistical data. Summary of the invention

[0005] The purpose of the present invention is to provide a method for compressing inter-layer feature maps of convolutional neural networks. Compared with the previous inter-layer feature map compression methods of convolutional neural networks, the present invention compresses the original inter-layer feature map data into non-zero element groups and data column index groups, and is expected to directly extract non-zero elements through the coordinates of non-zero elements in subsequent calculations, thereby skipping unnecessary calculation cycles. At the same time, the method is combined with improved incremental compression to compress non-zero values, which has a higher compression ratio than the compression method that only compresses zero values. In addition, the compression method of the present invention for non-zero values ​​does not rely on the correlation between data, but only requires the central trend index of statistical data, thereby reducing hardware resources and the overall amount of calculation.

[0006] The technical solution adopted by the present invention is: a method for compressing inter-layer feature maps of a convolutional neural network, comprising a software end and a hardware end;

[0007] The software end is used to count the median of the data exponent bits of the feature map between the convolutional neural network layers and calculate the threshold range of the data exponent bit compression;

[0008] The hardware end includes a sparse matrix row compression module, a redundant zero-marked element filtering module, a non-zero element incremental compression module and a decompression module; the sparse matrix row compression module is used to perform preliminary compression on the inter-layer feature map data, extract non-zero elements and record the corresponding column index, and record a zero-marked element when the data wraps;

[0009] The redundant zero-marked element filtering module removes the zero-marked element when the column index before the zero-marked element in the column index is greater than or equal to the column index after the zero-marked element;

[0010] The non-zero element incremental compression module compresses the exponent bit of the inter-layer feature map data to reduce the data bit width by using the median of the exponent bit obtained by the software end; and determines whether the non-zero element is incrementally compressed, and generates an incremental compression flag code;

[0011] The sparse matrix row compression module includes a non-zero element extraction unit, a column index generation unit and a row zero mark element unit; the non-zero element extraction unit is used to filter the non-zero elements in the inter-layer feature map data and pass them into the non-zero element incremental compression module; the column index generation unit generates a column index and passes the column index into the redundant zero mark element filtering module; the row zero mark element unit generates a row zero mark element and passes the row zero mark element into the redundant zero mark element filtering module;

[0012] The decompression module comprises a de-incrementation module and a de-indexing module;

[0013] The de-increment module finds the corresponding compressed data according to the incremental compression flag generated by the non-zero element incremental compression module, and then adds the exponent bit of the compressed data to the median of the counted exponent bits to decompress the non-zero data;

[0014] The de-indexing module finds the column index and row index of the original non-zero element according to the column index data generated by the redundant zero-marked element filtering module.

[0015] The compression step of the method of the present invention comprises:

[0016] Step 1, the software end uses Python software to extract the inter-layer feature map data of the convolutional neural network and extract the exponential bit of the data, and at the same time, statistically obtains the median of the data exponential bit and calculates the threshold range of the data exponential bit compression;

[0017] Step 2: passing the inter-layer feature map data of the convolutional neural network into the sparse matrix row compression module to generate a non-zero element set value_nozero and a column index element set col_index, where col_index includes the recorded line break zero mark element; passing value_nozero into the non-zero element incremental compression module, and passing col_index into the redundant zero mark element filtering module;

[0018] Step 3, the redundant zero-marked element filtering module compares the sizes of the elements before and after the zero-marked element in col_index, and if the previous element of the zero-marked element is smaller than the next element or one of the previous element or the next element is zero, the zero-marked element is retained, and a set col_index_zero including column index and zero-marked elements is generated;

[0019] Step 4, the non-zero element incremental compression module determines whether value_nozero needs to be compressed based on the median counted in the compression step 1 of the method of the present invention and the threshold: if the inter-layer feature map data is within the threshold range, the exponent bit is incrementally compressed, that is, the median is subtracted and the difference is retained as the exponent bit of the data; if the inter-layer feature map data is outside the threshold range, no compression operation is performed, and an incremental non-zero element set value_nozero_incre is generated. At the same time, the elements in value_nozero are encoded, and the element positions that perform incremental compression are recorded as "1", and the element positions that do not perform incremental compression are recorded as "0", and an incremental compression flag set incre_flag is generated; because the exponent bits of the inter-layer feature map data of the convolutional neural network are mostly concentrated in a certain range, only compressing the data within the threshold can still compress most of the inter-layer feature map data.

[0020] The decompression step of the method of the present invention comprises:

[0021] Step 1, perform incremental decompression, pass value_nozero_incre and incre_flag into the incremental decompression module, and the incremental compression module adds the exponent bit of the compressed data at the position recorded as "1" to the median in the compression step 1 of the method of the present invention to decompress the data according to the incremental compression element position recorded in incre_flag, and the compressed data at the position recorded as "0" remains unchanged;

[0022] Step 2: perform positional decompression, input the data decompressed in step 1 of the decompression method of the present invention and col_index_zero into the deindexing module, and the deindexing module adds one to the row count based on the size of the elements before and after the column index recorded in col_index_zero. If the previous element is larger than or equal to the next element, the row count is increased by one, or the row count is increased by one when a zero mark element is detected, thereby completing the operation of decompressing the data, data column index and data row index. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a diagram of a sparse compression device for feature maps between layers of a convolutional neural network;

[0024] Figure 2 This is a diagram of a sparse decompression device for feature maps between layers of a convolutional neural network;

[0025] Figure 3 The figure is a schematic diagram of sparse matrix compression storage; the right side is the compressed matrix storage format. Specific implementation plan

[0026] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0027] Step 1: Use Python software to analyze the trained convolutional neural network. First, input 100 images to be processed into the convolutional neural network, then extract the data of each convolutional layer output feature map, analyze the statistical information of the index bits of these data, and obtain the median med_num of the index bit, threshold data thre_value1 and thre_value2.

[0028] Step 2: Input the inter-layer feature map data inr_data as follows Figure 1 The sparse matrix row compression module shown in the figure uses a comparator to determine whether the input data is zero. If it is not zero, it is input to the next module. At the same time, the column index corresponding to the input data is generated. The sparse matrix row compression module also passes the number of columns col_num of the inter-layer feature map into the lookup table in advance. When the number of data num passed in reaches col_num, a line break flag "0" is recorded. At this time, the column index data and the line break flag form a set col_index.

[0029] Step 3: Input the non-zero element value_nozero obtained in step 2 of the specific implementation scheme into Figure 1 The non-zero element incremental compression module shown in the figure stores the med_num, thre_value1 and thre_value2 mentioned in step 1 of the specific implementation scheme in advance. When value_nozero is input, its exponent bit exp_bits is compared and judged, and the exponent bit is incrementally compressed within the range of greater than thre_value1 and less than thre_value2, that is, the exponent bit is subtracted from med_num to obtain the difference exponent bit diff_bits, and the position of the element to be incrementally compressed is recorded as "1", and the position not to be incrementally compressed is recorded as "0", and then the compressed data compress_data is composed of diff_bits and the mantissa bits frac_bits of the original input data, and the incremental compression flag set incre_flag is generated.

[0030] Step 4: Input the index data col_index obtained in step 2 of the specific implementation scheme into Figure 1 The redundant zero mark element filtering module shown detects the zero element in col_index, determines the size of the elements before and after the zero element, and retains the zero mark element if the element zero_fro before the zero element is smaller than the element zero_late after the zero element or zero_fro is zero or zero_late is zero. Otherwise, the mark element is removed and the next element is directly output. Finally, a set of column index plus zero mark elements, col_index_zero, is output.

[0031] Step 5: Decompress incremental compression and input compress_data and incre_flag into Figure 2 In the decompression module shown, if incre_flag is "1", the exponent bits diff_bits of the corresponding compressed_data are added to med_num to restore the original data exponent bits, and finally concatenated with the mantissa bits frac_bits to form the original non-zero data value_nozero.

[0032] Step 6: Unzip the position and input col_index_zero as Figure 2 The de-indexing module shown in the figure determines whether the input col_index_zero is zero. If col_index_zero is zero, the row index row_index is increased by one. If col_index_zero is not zero, the row index row_index remains unchanged. Determine the size of the previous and next elements in the input col_index_zero. If both the previous and next elements are not zero and the previous element is greater than or equal to the next element, then row_index is increased by one. If both the previous and next elements are not zero but the previous element is less than the next element, then row_index remains unchanged. In this way, the row index row_index of the original non-zero data is solved. Adding the col_index originally recorded by col_index_zero, the position of the non-zero elements of the inter-layer feature map data is solved.

[0033] Without considering incremental compression, data is stored as follows Figure 3 As shown on the right, the data is stored as non-zero values ​​and the column index of the data. The zero stored in the column index is a mark for when the matrix wraps. When decompressing, the program needs to read the elements in the column index matrix one by one. When it detects that the previous number of the current number is greater than or equal to the current number, it can also determine that the matrix wraps, thereby calculating the row index of the data. Subsequent calculations can be performed based on the solved data itself, the column index and the row index.

[0034] Example

[0035] In this embodiment, the inter-layer data of the VGG16 network data format of FP32 is compressed, and the 32-bit inter-layer data is set to be compressed to 26 bits, that is, the exponent bit length is reduced from 8 bits to 2 bits. The inter-layer feature map data of each convolutional layer is sequentially input to the compression device, and the pre-stated median and threshold of the inter-layer data are input. Since the data of each layer is different, the median and threshold will vary according to different inter-layer data, so that the overall compression effect will be improved. After the data is input to the compression device, two data streams are mainly output, one is the compressed data, which includes incremental compressed data and a mark for recording incremental compressed data, and the other is the column index record of the data, which includes the column index of non-zero data and the line break record value of the data. When the data is decompressed, the compressed data and the column index record of the data are input into the decompression device, and the decompressed data stream includes the original non-zero data, the column index of the data, and the row index for subsequent convolution calculations. Finally, the overall compression ratio of the inter-layer data of VGG16 FP32 is 2.683.

[0036] The above is only a specific embodiment of the present invention. It should be noted that the present invention is not limited to the specific form disclosed, nor does it exclude other possible embodiments. The present invention can be applied to various combinations, variations and environments, and can be adjusted and modified according to the above description or the technology and knowledge in the relevant field. However, as long as it does not depart from the spirit and scope of the present invention, any changes and improvements made by those skilled in the art should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method and device for sparse compression of inter-layer feature maps of a convolutional neural network, characterized in that: The device includes a software end and a hardware end; The software end is used to count the median of the data exponent bits of the feature map between the convolutional neural network layers and calculate the threshold range of the data exponent bit compression; The hardware end includes a sparse matrix row compression module, a redundant zero-marked element filtering module, a non-zero element incremental compression module and a decompression module; The sparse matrix row compression module is used to preliminarily compress the inter-layer feature map data, extract non-zero elements and record the corresponding column index, and record a zero mark element when the data wraps; The redundant zero-marked element filtering module removes the zero-marked element when the column index before the zero-marked element in the column index is greater than or equal to the column index after the zero-marked element; The non-zero element incremental compression module compresses the exponent bits of the inter-layer feature map data to reduce the data bit width by using the median of the exponent bits obtained by the software end statistics; And determine whether to perform incremental compression on non-zero elements, and generate incremental compression flag code; The sparse matrix row compression module includes a non-zero element extraction unit, a column index generation unit, and a row-break zero mark element unit; The non-zero element extraction unit is used to screen the non-zero elements in the inter-layer feature map data and pass them into the non-zero element incremental compression module; The generating column index unit generates a column index, and transmits the column index to the redundant zero mark element filtering module; the line break zero mark element unit generates a line break zero mark element, and transmits the line break zero mark element to the redundant zero mark element filtering module; The decompression module comprises a de-incrementation module and a de-indexing module; The de-increment module finds the corresponding compressed data according to the incremental compression flag generated by the non-zero element incremental compression module, and then adds the exponent bit of the compressed data to the median of the counted exponent bits to decompress the non-zero data; The de-indexing module finds the column index and row index of the original non-zero element according to the column index data generated by the redundant zero-marked element filtering module.

2. The method and device for sparse compression of inter-layer feature maps of convolutional neural networks according to claim 1, characterized in that The invention comprises a convolutional neural network inter-layer feature map sparse compression method and a convolutional neural network inter-layer feature map decompression method, wherein the convolutional neural network inter-layer feature map sparse compression method comprises the following steps: Step 1, the software end uses Python software to extract the inter-layer feature map data of the convolutional neural network and extract the exponential bit of the data, and at the same time, statistically obtains the median of the data exponential bit and calculates the threshold range of the data exponential bit compression; Step 2: passing the inter-layer feature map data of the convolutional neural network into the sparse matrix row compression module to generate a non-zero element set value_nozero and a column index element set col_index, where col_index includes the recorded line break zero mark element; passing value_nozero into the non-zero element incremental compression module, and passing col_index into the redundant zero mark element filtering module; Step 3, the redundant zero-marked element filtering module compares the sizes of the elements before and after the zero-marked element in col_index, and if the previous element of the zero-marked element is smaller than the next element or one of the previous element or the next element is zero, the zero-marked element is retained, and a set col_index_zero including column index and zero-marked elements is generated; Step 4, the non-zero element incremental compression module judges whether value_nozero needs to be compressed according to the median counted in step 1 of the convolutional neural network inter-layer feature map sparse compression method in combination with the threshold: if the inter-layer feature map data is within the threshold range, the exponent bit is incrementally compressed, that is, the median is subtracted and the difference is retained as the exponent bit of the data; if the inter-layer feature map data is outside the threshold range, no compression operation is performed, and an incremental non-zero element set value_nozero_incre is generated, and the elements in value_nozero are encoded, and the element positions that perform incremental compression are recorded as "1", and the element positions that do not perform incremental compression are recorded as "0", and an incremental compression flag set incre_flag is generated; The steps of the convolutional neural network inter-layer feature map decompression method include: Step 1, perform incremental decompression, pass value_nozero_incre and incre_flag into the incremental decompression module, and the incremental compression module adds the exponent bit of the compressed data at the position recorded as "1" to the median in step 1 of the convolutional neural network inter-layer feature map sparse compression method to decompress the data according to the incremental compression element position recorded in incre_flag, and the compressed data at the position recorded as "0" remains unchanged; Step 2: Perform position decompression, and input the decompressed data and col_index_zero in step 1 of the convolutional neural network inter-layer feature map decompression method into the deindexing module. The deindexing module adds one to the row count based on the size of the elements before and after the column index recorded in col_index_zero. If the previous element is larger than or equal to the next element, or if a zero mark element is detected, the row count is also added by one, thereby completing the decompression of the data, data column index, and data row index.

Images