Methods of data transfer and merging

Abstract

The present invention discloses a method for data transmission and merging, which is applicable to a sending end and a receiving end that are connected to each other for communication, and the method includes: a sending end stage and a receiving end stage. The sending end stage includes: transmitting first block data, second block data and third block data to a receiving end, obtaining fourth block data and fifth block data and transmitting the third block data, fourth block data and fifth block data to the receiving end. The receiving end stage includes receiving the first block data, the second block data and the third block data, merging the first block data, the second block data and the third block data to perform a convolution operation, receiving the fourth block data and the fifth block data, and merging the third block data, the fourth block data and the fifth block data to perform another convolution operation.

Description

Technical Field

[0001] The present invention relates to a convolutional neural network accelerator, and more particularly to a method for cutting data for transmission and merging data in a convolution operation of a flat processing. Background Art

[0002] Convolutional Neural Network (CNN) is currently considered one of the most widely used machine learning techniques in computer vision and image processing. The main operation of a convolutional neural network is the convolution between the kernel and the feature map, which consumes a lot of power through multiply accumulate (MAC) operations.

[0003] Compared with the energy waste of redundant operations, how to improve data access capabilities and reduce data transmission bandwidth is more important in future accelerator design. First, because the memory bandwidth grows slower than the processing speed of the processing unit, it means that the same algorithm may be limited by the memory and its architecture; second, because current neural networks mostly use small convolution kernels with deeper networks, which reduces MAC operations but increases memory usage. According to statistics, as the neural network model evolves, the power consumed by accessing feature maps on dynamic random access memory (DRAM) is more considerable than the power consumed by other operations.

[0004] Current CNNs usually use tiled processing, that is, the processing unit loads one block from the external memory each time for calculation. For example, the data blocks stored in the external memory DRAM are directly loaded into the static random access memory (SRAM) near the processing unit as a cache without compression. However, this method consumes a lot of power and occupies a lot of memory bandwidth when accessing DRAM each time the processing block is switched. For example, the data stored in the DRAM is cut into multiple sub-tensors of the same size and compressed, and then the compressed sub-tensors are sent to the SRAM for decompression, and the processing unit extracts the required block data from the SRAM for calculation. Although compressing block data can save power and bandwidth consumed during data transmission, if the sub-tensor cutting size is set too large, it may cause the SRAM to store data that will not be used in this processing, resulting in a waste of SRAM space. Or in order to obtain complete block data, it takes time to decompress a large file, but only a small amount of data is available. On the other hand, if the sub-tensor split size is set too small, in order to decompress and restore the complete block data in the correct order, additional bandwidth is required to load a large number of indicators to obtain the location of each compressed file. Summary of the invention

[0005] In view of this, the present invention proposes an efficient and hardware-oriented data access scheme suitable for CNN feature maps. The present invention divides data into subtensors of different sizes and uses a small number of indices to store these subtensors in a compressed but random access format. This design enables current CNN accelerators to instantly acquire and decompress subtensors in a flat processing manner. The present invention is applicable to data access architectures that require alignment and merging, and only minor modifications to the existing architecture are required to apply the present invention.

[0006] A method for data transmission and merging according to an embodiment of the present invention is applicable to a sending end and a receiving end that are connected to each other for communication, and the method includes: a sending end stage and a receiving end stage. The sending end stage includes: transmitting the first block data, the second block data and the third block data to the receiving end, obtaining the fourth block data and the fifth block data and transmitting the third block data, the fourth block data and the fifth block data to the receiving end. The receiving end stage includes receiving the first block data, the second block data and the third block data, merging the first block data, the second block data and the third block data to perform a convolution operation, receiving the fourth block data and the fifth block data, and merging the third block data, the fourth block data and the fifth block data to perform another convolution operation.

[0007] In summary, the present invention proposes an effective storage scheme for input feature maps, which can reduce the bandwidth of external memory and conform to the memory access mode in the existing CNN accelerator architecture. Given a specific CNN layer and accelerator configuration, the present invention can cut tensor data into multiple sub-tensors of specific sizes. The existing CNN accelerator can integrate the present invention with a small amount of hardware modification to improve the overall performance.

[0008] The above description of the contents of the specification of the present invention and the following description of the implementation modes are used to demonstrate and explain the spirit and principle of the present invention, and to provide a further explanation of the scope of the patent application of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a flow chart of a method for data transmission and merging according to an embodiment of the present invention;

[0010] Figure 2 It is a schematic diagram of the feature map being cut into multiple data blocks in the horizontal direction;

[0011] Figure 3 is a schematic diagram of a first embodiment;

[0012] Figure 4 It is a cutting diagram of the input data;

[0013] Figure 5 An example of a configuration method of the present invention is shown for general convolution;

[0014] Figure 6 An example of another configuration of the dilated convolution according to the present invention is shown; and

[0015] Figure 7 An example of subtensor and index storage in the present invention is shown.

[0016] Wherein, the reference numerals are:

[0017] P1 sender stage

[0018] P2 receiving stage

[0019] Steps S1 to S7

[0020] A1 Starting address of sub-tensor 1

[0021] A5 The starting address of subtensor 5

[0022] SZ1~SZ8 data size of sub-tensors 1~8 DETAILED DESCRIPTION

[0023] The detailed features and characteristics of the present invention are described in detail in the following embodiments, and the content is sufficient for those skilled in the art to understand the technical content of the present invention and implement it accordingly. According to the content disclosed in this specification, the claims and the drawings of the specification, those skilled in the art can easily understand the relevant concepts and characteristics of the present invention. The following examples further illustrate the viewpoints of the present invention in detail, but do not limit the scope of the present invention in any viewpoint.

[0024] The present invention is applicable to any field with convolution operation. In the present invention, a method for data transfer set merging is proposed, which includes an input feature map segmentation method to avoid accessing partially compressed subtensors and minimize the number of subtensors to avoid data fragmentation.

[0025] Figure 1 The flowchart of the method of data transmission and merging according to one embodiment of the present invention is applicable to the sending end and the receiving end of the communication connection. The sending end, for example, includes an external memory DRAM and a control circuit for processing block data segmentation, and the receiving end, for example, includes a CNN accelerator including a computing unit and a cache SRAM.

[0026] Figure 2 It is a schematic diagram of the input feature map being cut into multiple data blocks in the horizontal direction. Assuming that the CNN accelerator processes one input feature map F each time, then when the input feature map F is processed for the i-1th time i-1 , processing the input feature map F for the i-th time i , and the input feature map F is processed for the i+1th time i+1

[0027] like Figure 1 As shown, the data transmission and merging method of an embodiment of the present invention is divided into a sending end stage P1 and a receiving end stage P2. The sending end stage P1 includes steps S1, S2 and S3. The receiving end stage P2 includes steps S4, S5, S6 and S7.

[0028] Step S1 is "the sending end transmits the first indicator, the first block data B1, the second block data B2 and the third block data B3 to the receiving end". Step S1 corresponds to the aforementioned "in the i-th processing of the input feature map F i In practice, before the sending end transmits the first block data B1, the second block data B2 and the third block data B3, it further includes using a compressor to compress the first block data B1, the second block data B2 and the third block data B3 respectively, thereby reducing the bandwidth occupied when transmitting to the receiving end. The first indicator is used to indicate the starting address of the first block data B1, the data size of the first block data B1, the data size of the second block data B2 and the data size of the third block data.

[0029] Step S2 is "the transmitting end obtains the fourth block data B4 and the fifth block data B5". In detail, the control circuit of the transmitting end can perform the following configuration on three consecutive input feature maps F adjacent to each other in space. i-1 、F i and F i+1 The data is divided to obtain first to fifth block data B1 to B5.

[0030] Step S3 is "the sending end transmits the second indicator, the third block data B3, the fourth block data B4 and the fifth block data B5 to the receiving end". Step S3 corresponds to the aforementioned "processing the input feature map F for the i+1th time i+1 In practice, before the sending end transmits the third block data B3, the fourth block data B4 and the fifth block data B5, it further includes using a compressor to compress the third block data B3, the fourth block data B4 and the fifth block data B5 respectively, thereby reducing the bandwidth occupied when transmitting to the receiving end. The second indicator is used to indicate the starting address of the third block data B3, the data size of the third block data B3, the data size of the fourth block data B4 and the data size of the fifth block data B5.

[0031] Step S4 is "the receiving end receives the first indicator, the first block data, the second block data and the third block data". Figure 1 As shown, step S4 is performed after step S1 is completed.

[0032] Step S5 is "the receiving end combines the first block data, the second block data and the third block data according to the first indicator to perform a convolution operation". In practice, before performing the convolution operation, it further includes decompressing the first block data B1, the second block data B2 and the third block data B3 with a decompressor. After the three block data B1-B3 are decompressed, they are stored in the SRAM. The processing unit can obtain the first starting address of the first block data B1 in the SRAM according to the first indicator, and calculate the second starting address of the second block data B2 in the SRAM according to the first starting address and the data size of the first block data B1, and then calculate the third starting address of the third block data B3 in the SRAM according to the second starting address and the data size of the second block data B2.

[0033] Step S6 is "the receiving end receives the second indicator, the third block data, the fourth block data, and the fifth block data". Figure 1 As shown, step S6 is performed after step S3 is completed.

[0034] Step S7 is "the receiving end combines the third block data, the fourth block data and the fifth block data according to the second indicator to perform another convolution operation". In practice, before performing the convolution operation, it further includes decompressing the third block data B3, the fourth block data B4 and the fifth block data B5 with a decompressor. After the three block data B3-B5 are decompressed, they are stored in the SRAM. The processing unit can obtain the third starting address of the third block data B3 in the SRAM according to the second indicator, and calculate the fourth starting address of the fourth block data B4 in the SRAM according to the third starting address and the data size of the third block data B3, and then calculate the fifth starting address of the fifth block data B5 in the SRAM according to the fourth starting address and the data size of the fourth block data B4.

[0035] like Figure 2 As shown, the present invention proposes a method for cutting the input feature map F i-1 、F i and F i+1 In the following, the implementation of the cutting configuration is further described in detail through two embodiments. The first embodiment illustrates the operation of the present invention with actual numbers, and the second embodiment illustrates the generalized implementation of the present invention in algebraic form.

[0036] Figure 3 It is a schematic diagram of the first embodiment. Assume that the CNN architecture is as follows: the convolution kernel size is 3×3 and the output feature map size is 8×8 blocks, and zero-padding is used to maintain the output feature map and the input feature map size to be the same.

[0037] In the first processing, a 10×10 input patch is extracted from the upper left corner of the input feature map. Figure 1 As shown, the horizontal left boundary is -1 and the right boundary is 9.

[0038] During the second processing, step 8 units to the right from the right edge of the first input block to extract the next input block.

[0039] Since the step length is constant in the same layer of CNN processing, the left and right boundaries of the input block extracted each time can form two arithmetic progressions. l ={-1, 7, 15, ...}, the right boundary is represented by B r = {9, 17, 25, ...}. The configuration proposed by the present invention is the division formed by these two boundaries. In other words, the configuration proposed by the present invention is the union G = B l ∪B r , in this example, G = {1, 7} (mod 8).

[0040] Because 7-1=6 (mod 8) and 1-7=2 (mod 8), and the above configuration is applicable to both horizontal and vertical cutting, each input feature map can be divided into sub-tensors of the following four shapes: 6×6, 2×6, 6×2, and 6×2. Figure 4 It is a schematic diagram of cutting the input data.

[0041] A 10×10 input block can be composed of one 6×6, two 2×6, two 6×2, and four 6×2.

[0042] In addition, since the access to halo data is limited to a two-dimensional plane, the segmentation configuration proposed by the present invention does not need to be implemented in the dimension of channels.

[0043] The second embodiment is described below. In this embodiment, the operation of each layer of CNN is generalized and represented by the following parameters:

[0044] The convolution kernel size is expressed as 2k+1, because the convolution kernel size is usually an odd number.

[0045] The stride is denoted by S. The stride is the distance between two elements that are convolved with two convolution windows. When s>1, the output feature map size is smaller than the input feature map size and the computational cost decreases.

[0046] The dilation distance is represented by d. In dilated convolution, two adjacent elements of the original convolution kernel are dilated into a convolution window according to this dilation distance.

[0047] The size of the output feature map is t h ×t w express.

[0048] Based on the above parameters, Figure 5 It shows that a general convolution adopts a configuration method of the present invention (k, s, d, t w )=(1,2,1,6).

[0049] When calculating the leftmost output element in the output feature map, the convolution window starts to extract from the leftmost side of the input feature map. Assuming that the leftmost boundary of the input feature map is -k, the rightmost boundary is (t w -1)s+k+1. The offset between two adjacent sub-tensors is st w The configuration method defined in the present invention is as follows:

[0050] G = {-k, (t w -1)s+k+1}(mod st w )

[0051] ={-k, k-s+1} (mod stw )

[0052] Figure 6 It shows that the dilated convolution uses another configuration of the present invention (k, s, d, t w )=(1,1,2,6).

[0053] Referring to the above operation method, another configuration method of dilated convolution can be obtained as follows:

[0054] G = {-kd, kd-s+1} (mod st w )

[0055] It can be seen from the configuration method proposed in the present invention that if N' divides N, then the configuration that complies with (mod N) also complies with (mod N').

[0056] Take AlexNet CONV1 as an example, its (k, s, t w )=(5, 4, 8), corresponding to a configuration G1={27, 2} (mod 32) of the present invention. Therefore, AlexNet CONV1 is also applicable to another configuration G2={3, 2} (mod 8) of the present invention.

[0057] As mentioned above, selecting a single N value in all convolutional layers can simplify hardware implementation. In one embodiment of the present invention, N=8 is a setting value suitable for most situations.

[0058] Given a cutting configuration of the present invention, the multiple sub-tensors cut out need to be stored in a data structure and meet the requirements of memory alignment to maximize the benefits of compression. Because the sizes of the sub-tensors after compression are different, the present invention additionally stores indicators for representing these sub-tensors.

[0059] Figure 7 Demonstrates how the present invention stores sub-tensors and indices. When setting indices, for adjacent sub-tensors, such as Figure 7 For sub-tensors 1, 2, 3, and 4, only the pointer A1 is used to indicate the starting address of block 1 and the pointers SZ1 to SZ4 are used to indicate the compressed data size of each of the four sub-tensors. Therefore, accessing these sub-tensors is divided into two steps: first, the starting address is obtained from the pointer A1, and then the size of each sub-tensor is obtained from the pointers SZ1 to SZ8 and accumulated to the starting address to obtain the actual offset of each sub-tensor.

[0060] Since the index of the present invention does not need to correspond to each sub-tensor, the total size of the index can be effectively reduced.

[0061] The present invention proposes a hardware-friendly data segmentation method that can be used for storage, access and compression of input feature maps. The present invention cuts the input feature map into multiple different sizes to avoid the receiver extracting unused sub-tensors and wasting cache space. The present invention only requires a small number of indicators to record the position of each compressed sub-tensor. The application only requires slight modifications to the existing CNN accelerator architecture, because the present invention is applicable to almost all compression algorithms and only requires changing the existing feature map cutting method. The present invention can save a large amount of memory transmission bandwidth.

[0062] In summary, the present invention proposes an effective storage scheme for sparse feature maps, which can reduce the bandwidth of external memory and conform to the memory access mode in modern CNN accelerator architecture. Given a specific CNN layer and accelerator configuration, the present invention can cut tensor data into multiple sub-tensors of specific sizes. Existing CNN accelerators can integrate the present invention with a small amount of hardware modification to improve overall performance.

Claims

1. A method for data transmission and merging, applicable to a sending end and a receiving end of a communication connection, characterized in that: The method comprises: The first sending stage includes: The sending end transmits a first block of data, a second block of data and a third block of data to the receiving end; Obtaining a fourth block of data and a fifth block of data by the sending end; and transmitting the third block data, the fourth block data and the fifth block data to the receiving end by the transmitting end; and A receiving end stage, including: Receiving the first block data, the second block data and the third block data by the receiving end; Combining the first block data, the second block data and the third block data with the receiving end to perform a convolution operation; receiving the fourth block of data and the fifth block of data by the receiving end; and The receiving end combines the third block data, the fourth block data and the fifth block data to perform another convolution operation.

2. The method for data transmission and merging as claimed in claim 1, characterized in that: When the sending end transmits the first block data, the second block data and the third block data to the receiving end, the method further comprises: transmitting a first pointer to the receiving end by the sending end, wherein the first pointer is used to indicate the starting address of the first block data, the data size of the first data block, the data size of the second data block and the data size of the third data block; and When the sending end transmits the third block data, the fourth block data and the fifth block data to the receiving end, the method further includes: transmitting a second pointer to the receiving end by the sending end, wherein the second pointer is used to indicate the starting address of the third block data, the data size of the third data block, the data size of the fourth data block and the data size of the fifth data block.

3. The method for data transmission and merging as claimed in claim 1, characterized in that: Before transmitting the first block data, the second block data and the third block data to the receiving end by the transmitting end, the method further comprises: compressing the first block data, the second block data and the third block data by a compressor; Before transmitting the third block data, the fourth block data and the fifth block data to the receiving end by the transmitting end, the method further comprises: compressing the third block data, the fourth block data and the fifth block data by the compressor; Before the receiving end combines the first block data, the second block data and the third block data to perform the convolution operation, the method further includes: decompressing the first block data, the second block data and the third block data by a decompressor; and Before the receiving end combines the fourth block data, the fifth block data and the sixth block data to perform the convolution operation, the method further includes: decompressing the fourth block data, the fifth block data and the sixth block data by the decompressor.

Images