An efficient neural network preprocessing method

Abstract

The present disclosure discloses a neural network preprocessing method, characterized in that: if m points of n channel data are required for each convolution, the rearrangement is as follows: in the first column, m*n numbers related to the first point to the mth point are arranged from bottom to top; further, if the convolution step is k, when the rearrangement continues to the second column from left to right: the data of n channels of the k+1th point is arranged from bottom to top; then, the data of n channels of the k+2th point to the k+mth point is arranged from bottom to top; further, when the rearrangement is to the i+1th column, n channel data of the i*k+1th point to the i*k+mth point is arranged from bottom to top. The present disclosure has a wide range of applications and can be used in various network acceleration schemes and various processors to improve memory utilization and reduce image convolution calculation time.

Description

Technical Field

[0001] This disclosure belongs to the field of image convolution calculation technology, and specifically relates to an efficient neural network preprocessing method. Background Technology

[0002] In deep learning, to improve data transmission bandwidth and computational performance, data formats such as NCHW, NHWC, and CHWN are used. These are logical data formats for images or feature maps (the order in which data is stored in memory). Different data formats can increase the reusability of local data and improve the utilization of data in the cache.

[0003] Figure 1 This illustrates the image convolution calculation decomposition process in the prior art, using... Figure 1 Taking a 3x3 convolution with a stride of 2 as an example, it shows that each convolution requires three points of three-channel data, but the stride is 2, just as... Figure 1 The gray area shown changes according to the step size. After calculating 0, 1, 2, it becomes 2, 3, 4. Because the step size is 2, it results in... Figure 1 The region represented by the gray 2 is reused. During convolution, the pixel addresses of the image become discontinuous with each convolution, leading to reduced memory utilization and frequent memory reads and writes, resulting in longer processing times. This is especially noticeable when observing... Figure 1 As shown in the lower part of the single-line input image, it can be clearly seen that: due to the convolution stride, the convolution calculation can reuse some channel data, but this will also cause the input image addresses of the two calculations to be discontinuous, resulting in branches during data scheduling and causing D Cache Miss (data cache miss). In the actual application of neural network accelerators, this will lead to a huge resource and time overhead.

[0004] To further improve the utilization rate of input image data, it is necessary to continuously improve the technology of data format and propose new data formats, while also considering the format conversion from known data formats to new data formats. Summary of the Invention

[0005] In view of this, the present disclosure provides a neural network preprocessing method, characterized in that:

[0006] If each convolution requires m points and n channels of data, then during rearrangement:

[0007] In the first column, first arrange the m*n numbers involved from the first point to the mth point from bottom to top;

[0008] Furthermore, if the convolution stride is k, then when rearranging from left to right to the second column: the data of the n channels of the (k+1)th point are first arranged from the bottom; then, the (k+2)th point is arranged sequentially from bottom to top, all the way up to the (k+m)th point;

[0009] Furthermore, when rearranging to the (i+1)th column, the data from the n channels of the i*k+1, i*k+2, ..., i*k+m points are arranged from bottom to top.

[0010] Preferred,

[0011] If each convolution requires N data points, where N corresponds to the computational parallelism of the neural network accelerator, then when N is greater than m*n channels of data, each column from the first column to the second-to-last column can be padded with N-(m*n) zeros.

[0012] Preferred,

[0013] For the last column, add NM zeros based on the actual number M of channel data in the last column.

[0014] Preferred,

[0015] The HWC data format conversion method is as follows during rearrangement, including the following steps:

[0016] S100: Each time, a whole line of data is read from the DDR main memory in the vertical direction. This is the data for one format conversion.

[0017] S200: Store all data sequentially into 8 storage areas. Each Block Mem storage area stores CH*W / 8 data items, where CH represents the number of channels and W represents the width.

[0018] S300: Reads data from 8 Block Mems in parallel, with each Block Mem outputting 8 bytes of data. The data is then concatenated into 64 bytes and stored back into DDR main memory.

[0019] Preferred,

[0020] The HWC data format conversion method is as follows during rearrangement, including the following steps:

[0021] S1: Read image data from DDR main memory. The original image data is stored in main memory in HWC format by default. The image is 256*256, so in each cycle, 16 points of 4-channel data values ​​can be read from DDR.

[0022] S2: The read data is processed using the aforementioned neural network preprocessing method to achieve channel rearrangement and channel compensation, where 0*4 represents the 4 compensated channels 0, and the width of each line of data after compensation is 128.

[0023] S3: Store the rearranged data into 8 FIFO first-in-first-out memories. The input width of each FIFO is 16 bytes and the output width is 8 bytes. Each FIFO stores 128 / 8 = 16 data points. The first FIFO stores the first 16 columns corresponding to points 0-15, and the last FIFO stores the last 16 columns of data.

[0024] S4: The previous 8 FIFOs were written sequentially. This step reads them in parallel, that is, reads the data of the 8 FIFOs at the same time. For the first FIFO: since the output bit width is only 8 bytes, the 16 bytes of data written in the previous cycle will be read out in two cycles. The first cycle reads BGRX(0-1), and the second cycle reads BGRX2 and 4 0s. For the second FIFO, the data read in the first cycle is GBRX(32-33), and the data read in the first cycle of the eighth FIFO is GBRX(224-225).

[0025] S5: The 8-byte data output from the 8 FIFOs is concatenated into 64 bytes and written back to the DDR main memory. When all the data in the 8 FIFOs is written out, the format conversion of the image from HWC to HWC88 after channel rearrangement is completed.

[0026] Thus, when all 256 rows of input image data have completed the five steps from S1 to S5, the entire original 4-channel 256*256 pixel image has also completed all rearrangement processing.

[0027] Preferred,

[0028] The HWC88 storage format is used to implement FM's reordering of data storage organization format, and it needs to meet the following conditions: the format has good locality, can improve the utilization of data in the cache, and improve data transmission bandwidth and computing performance.

[0029] Preferred,

[0030] The HWC88 storage format is specifically as follows:

[0031] Divide a row or a portion of a row of FM data into 8 blocks. Place the data of the same channel group at the same position in each block at the same memory address, prioritizing the storage of all channels of the same pixel in each block.

[0032] Preferred,

[0033] FM uses the HWC88 format for convolution operations, specifically by dividing the previous row of FM data into 8 equal blocks to obtain 8 parallel points, and multiplying the data of the same position and channel group in each block with the corresponding channel of the convolution kernel.

[0034] Therefore, this disclosure proposes an efficient neural network preprocessing method, which implements a rearrangement scheme for input images in neural network computation. Furthermore, this disclosure implements the conversion from the existing HWC format to the new format (i.e., HWC88 format) required for this more efficient scheme. This facilitates the development of a new neural network accelerator based on the scheme disclosed in this disclosure. It should be noted that the HWC88 storage format is used to implement the rearrangement of the FM organization format in data storage, and must meet the following conditions: this format has good locality, can improve the utilization of data in the cache, and improve data transfer bandwidth and computational performance. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of image convolution calculation decomposition in existing technology;

[0036] Figures 2A to 2C This is a single-row channel rearrangement diagram in one embodiment of this disclosure;

[0037] Figure 3 This is a schematic diagram of channel rearrangement at 80 points in one embodiment of this disclosure;

[0038] Figure 4 This is a conversion diagram of HWC format and HWC88 format data in one embodiment of this disclosure;

[0039] Figure 5 This is a schematic diagram illustrating data format conversion via a storage area in one embodiment of this disclosure;

[0040] Figure 6 This is a schematic diagram of RGB image format conversion for 80 points in one embodiment of this disclosure;

[0041] Figure 7A This is a schematic diagram of channel rearrangement at 256 points in one embodiment of this disclosure;

[0042] Figure 7B This is a schematic diagram of channel rearrangement at 256 points in another embodiment of this disclosure;

[0043] Figure 8 This is a diagram illustrating the storage format of HWC88 format FM data in one embodiment of this disclosure;

[0044] Figure 9 This is an example diagram of a basic HWC88 in one embodiment of this disclosure;

[0045] Figure 10 This is an example diagram of segmenting a single line of the entire input data fm in one embodiment of this disclosure;

[0046] Figure 11 This is an example diagram of an improved HWC88 in one embodiment of this disclosure;

[0047] Figure 11A This is a schematic diagram of the HWC88 storage format for multiple batches in one embodiment of this disclosure;

[0048] Figure 12 This is a diagram showing the storage format of WT data when FM performs convolution operations using the HWC88 format in one embodiment of this disclosure;

[0049] Figure 13 This is an example diagram of the hardware implementation of a neural network preprocessing method in one embodiment of this disclosure;

[0050] Figure 14 This is based on one embodiment of the present disclosure. Figure 13 The diagram illustrates the hardware architecture and provides an example of the data processing flow during rearrangement. Detailed Implementation

[0051] The various embodiments of this disclosure will be described in detail below.

[0052] against Figure 1 The existing diagram of image convolution calculation decomposition in the prior art has problems. For the three color channels of three pixels in each row, in order to further improve the computational parallelism and memory utilization, this disclosure proposes an efficient neural network preprocessing method to realize a channel rearrangement scheme for input image data.

[0053] See Figures 2A to 2C This illustrates the single-lane lane rearrangement disclosed herein:

[0054] Taking single-row data as an example, the single-row data is from... Figure 2B The format shown is converted to Figure 2C The format shown is illustrated below. For example, a convolution stride of 2 is used. Please note... Figure 2A , Figure 2B , Figure 2C The CH -> arrow direction, H -> arrow direction, W -> arrow direction, and the three colors R, G, and B are described below:

[0055] The neural network preprocessing method is characterized by:

[0056] During image channel rearrangement, the original Figure 2A , Figure 2BThe data from the three channels (0, 1, 2) of points 0 to 39 in the R, G, and B coordinates shown in the image are rearranged from left to right. Figure 2C The image shows the 9 channels of a point from bottom to top (where 3 times 3 equals 9, and is represented from bottom to top as 0, 0, 0, 1, 1, 1, 2, 2, 2). Furthermore, since the convolution stride is 2, the data of the original second pixel is further rearranged into three-channel data for pixels 2, 3, and 4 (represented from bottom to top as 2, 2, 2, 3, 3, 3, 4, 4, 4). Thus... Figures 2A to 2C This diagram illustrates all the input image -> single-line input image -> single-line input image channel rearrangements.

[0057] Will Figure 2C and Figure 1 By comparison, we can see that the nine channels at each point after rearrangement are exactly the same as the three-channel data from the original three points needed for each convolution. We can also see that because the stride in this embodiment is 2, and the sequence from 0, 0, 0 to 1, 1, 1 is equivalent to two three-channel sequences, therefore... Figure 2C The following characteristics are shown: 2, 2, 2; and the subsequent 4, 4, 4, etc. are all repeated once.

[0058] Therefore, for the three color channels of the three pixels in each row, the neural network preprocessing method disclosed in this disclosure can improve computational parallelism and memory utilization.

[0059] Furthermore, it is understandable that, Figures 2A to 2C Taking the illustrated embodiment as an example, the key to this neural network preprocessing method lies in: firstly, each convolution requires n-channel data down to m points (e.g., Figure 1 , Figures 2A to 2C All data consists of 3 points and 3 channels (e.g., R, G, B), which determines the second factor: the convolution stride k.

[0060] Let's combine the above... Figures 2A to 2C It can be found that:

[0061] If each convolution requires m points and n channels of data, then during rearrangement, it's similar to... Figure 2C so:

[0062] In the first column, first arrange the m*n numbers involved from the first point to the mth point from bottom to top;

[0063] Furthermore, if the convolution stride is k, then it is similar to... Figure 2C Thus, when rearranging from left to right to the second column: the data of the n channels at the (k+1)th point are arranged starting from the bottom; just as... Figure 2CLike the 2, 2, 2 pattern of the third point; then, arrange the (k+2)th point sequentially from bottom to top, all the way up to the (k+m)th point;

[0064] ...

[0065] Furthermore, when rearranging to the (i+1)th column, the data from the n channels of the i*k+1, i*k+2, ..., i*k+m points are arranged from bottom to top.

[0066] In another embodiment,

[0067] For example, a single grayscale channel can be rearranged into 8 channels, where 3 channels are data from 3 single-channel pixels and 5 channels are padded with 0s; for a 4-channel image, the data from 3 4-channel pixels reaches 12, and the nearest integer multiple of 8 is 16 channels. Therefore, it can be rearranged into 16 channels, where 12 channels are data from 3 4-channel pixels and 4 channels are padded with 0s.

[0068] By extension, in this disclosure, the stride and kernel size of the convolution are both scalable, such as:

[0069] The kernel size is 4*4. Taking a 3-channel RGB image as an example, a point after channel rearrangement will have 12 channels, containing three-channel data from 4 points. To compensate it to the nearest integer multiple of 8 channels, i.e., 16 channels, only 4 channels need to be filled with 0.

[0070] Assuming the step size changes:

[0071] When the step size was originally 2, the first point after rearrangement contained the data of three pixels: 0, 1, and 2. Now, with a step size of 1, the second point contains the data of pixels 2, 3, and 4.

[0072] If the step size is changed to 1, then the 0, 1, and 2 of the first point remain unchanged. The corresponding change is that the pixel data of the second point changes from 2, 3, and 4 to 1, 2, and 3, and so on for all subsequent points.

[0073] It should be noted that the zero-padding step can be used in the neural network preprocessing method disclosed in this publication. If each convolution requires N data points (note: N corresponds to the computational parallelism of the neural network accelerator), when N is greater than m*n channels, then from the first column to the second-to-last column, each column should be padded with N-(m*n) zeros; for the last column, based on the actual number M of channels in the last column, NM zeros should be padded. It is understood that the actual number M of channels in the last column may be equal to or less than the second-to-last column. Further details regarding zero-padding in the last column can be found below. Figure 3 The examples revealed.

[0074] The above describes a neural network preprocessing method disclosed in this disclosure. It can be observed that it is not limited to the number of channels or the convolution stride. Furthermore, it is not limited to the convolution kernel size. This disclosure clearly proposes a rearrangement method for input images, including image rearrangement and image format conversion without limiting the number of channels, as well as rearrangement of input images under various convolution kernel sizes and different convolution strides. This means that this disclosure has a wide range of applications and can be used in various network acceleration schemes and various processors to improve memory utilization and reduce image convolution computation time.

[0075] The following detailed examples illustrate image format conversion schemes.

[0076] For example, suppose we need to develop a neural network accelerator HiPU with a computational parallelism of 8, and assume it stores all data in HWC88 format instead of the existing HWC format. This means that the design parameters of the HiPU neural network accelerator include: since the HiPU's computational parallelism is 8, the number of channels needs to be compensated to be an integer multiple of 8, and the image data format needs to be converted from HWC to HWC88.

[0077] Taking an RGB input image with 80 pixels per row as an example, after rearranging the input image channels, each row has 40 pixels, and each pixel has 9 channels. Because it is necessary to further compensate the number of channels to be a multiple of 8, the number of channels is padded to 16, and the 7 compensated channels are all set to 0. Specifically, as shown below... Figure 3 As shown. Figure 3 This illustrates the channel rearrangement at 80 points. Figure 3 In Chinese, the HWC format is also referred to as HWC8.

[0078] Taking 40 channels of input data with a width of 16 as an example, Figure 4 This refers to the corresponding arrangement of data stored in HWC8 and HWC88 formats respectively.

[0079] This example illustrates the general conversion method between HWC and HWC88.

[0080] S100: Reads a complete line of data vertically from DDR main memory each time; this is data for one format conversion. Figure 4 In the example, the height of the original data is only one row. At this time, one row of data is all the data of 16 points and 40 channels in the figure.

[0081] S200: All data is sequentially stored into 8 storage areas. Each Block Mem storage area stores CH (number of channels) * W (width) / 8 data items. Figure 5 As shown.

[0082] S300: Reads data from 8 Block Mems in parallel, with each Block Mem outputting 8 bytes of data. The data is then concatenated into 64 bytes and stored back into DDR main memory.

[0083] Combining the channel compensation and format conversion sections above, let's take an RGB input image with 80 pixels per row as an example. The complete image format conversion is as follows: Figure 6 As shown.

[0084] For example, in the following embodiment:

[0085] Taking an input image of 256x256, 4-channel RGBX, and a convolution stride of 2 as an example, an embodiment of this disclosure is described in detail.

[0086] First, in this example, the data size of each channel is one byte, and the DDR main memory used in this example reads and writes 64 bytes of data per cycle.

[0087] S1: Read image data from DDR main memory. Since the original image is stored in main memory in HWC format by default, 16 points of 4-channel data values ​​can be read from DDR in each cycle.

[0088] S2: Perform channel rearrangement and channel compensation on the read data. The processed data is as follows: Figure 7A As shown. Here, 0*4 represents the four compensated channels 0, and the width of each row of data after compensation is 128.

[0089] S3: Store the rearranged data sequentially into eight FIFOs (First-In, First-Out) memory. Each FIFO has a 16-byte input width and an 8-byte output width. Each FIFO stores 128 / 8 = 16 data points. That is, the first FIFO stores points 0-15. Figure 7A , Figure 7B The first 16 columns. The last FIFO stores the last 16 columns of data;

[0090] S4: The previous eight FIFOs were written sequentially; step 4 reads them in parallel. That is, data from all eight FIFOs is read simultaneously. Taking the first FIFO as an example, since the output bit width is only 8 bytes, the 16 bytes of data written in the previous cycle will be read out in two cycles. The first cycle reads BGRX (0-1), and the second cycle reads BGRX2 and four 0s. The data read from the second FIFO in the first cycle is GBRX (32-33), and the data read from the eighth FIFO in the first cycle is GBRX (224-225).

[0091] S5: The 8-byte data output from these 8 FIFOs is concatenated into 64 bytes and written back to the DDR main memory. When all data in the 8 FIFOs is written out, the format conversion from HWC to HWC88 after channel rearrangement is completed.

[0092] Thus, when all 256 rows of input image data have completed the above 5 steps, the entire original 4-channel 256*256 pixel image has completed the rearrangement process.

[0093] In another embodiment, compared to Figure 7A For example, taking an input image of 256x256, 4-channel RGBX, and a convolution stride of 1, Figure 7B The scalability of this disclosure under varying convolution stride is also described. Firstly, the only difference between this example and the previous one is the convolution stride; the difference in channel rearrangement results caused by the convolution stride is as follows: Figure 7A , Figure 7B As shown, Figure 7A This is the result when the convolution stride is 2, as in the previous example. Figure 7B This is the result when the convolution stride becomes 1.

[0094] Since the step size is reduced to 1, to keep the input and output image sizes unchanged, padding needs to be added before and after the image. The unchanged image size also means that W after image rearrangement is still 256. The fact that W remains 256 after image rearrangement implies a change in the instantiated FIFO depth. If the FIFO input bit width is 16B and the FIFO depth is 32, then if the FIFO input bit width is 8B and the FIFO depth is 64, the output bit width will also be 8B.

[0095] In fact, this disclosure is highly scalable:

[0096] 1. The number of channels in the input image is scalable.

[0097] For example, a single grayscale image can be reordered into 8 channels, where 3 channels are data from 3 single-channel pixels and 5 channels are padded with zeros. For a 4-channel image, it can be reordered into 16 channels, where 12 channels are data from 3 4-channel pixels and 4 channels are padded with zeros.

[0098] 2. The stride and kernel size of convolution are both scalable.

[0099] For example, if the convolution kernel size is 4*4, taking a 3-channel RGB image as an example, then after channel rearrangement, a single point will have 12 channels, containing the three-channel data of 4 points. To compensate it to the nearest integer multiple of 8 channels (16 channels), only 4 zero channels need to be added. Assuming the stride changes, if the stride was originally 2, after re-taking, the first point contains the data of three pixels: 0, 1, and 2, and the second point contains the data of 2, 3, and 4. If the stride is changed to 1, then the 0, 1, and 2 of the first point remain unchanged, but the pixel data contained in the second point changes from 2, 3, and 4 to 1, 2, and 3, and so on for all subsequent points.

[0100] 3. The dilation of convolutional kernels is scalable.

[0101] The change in the dilation of the convolution kernel can be achieved simply by loading the corresponding IMG data at the channel rearrangement. This will only result in the load IMG address being discontinuous, but it still confirms that the proposed scheme has scalability in the dilation of the convolution kernel.

[0102] Obviously, when the neural network preprocessing method disclosed in this disclosure performs rearrangement, it can include image rearrangement and image format conversion with unlimited number of channels, as well as rearrangement of input images with various convolution kernel sizes and different dilations under different convolution strides.

[0103] Regarding the HWC88 storage format disclosed in this disclosure, it is used to implement the rearrangement of the organization format in data storage by FM, and needs to meet the following conditions: the format has good locality, can improve the utilization of data in the cache, and improve data transmission bandwidth and computing performance.

[0104] For example, the HWC88 storage format is specifically as follows:

[0105] Divide a row or a portion of a row of FM data into 8 blocks. Place the data of the same channel group at the same position in each block at the same memory address, prioritizing the storage of all channels of the same pixel in each block.

[0106] For example, FM uses the HWC88 format for convolution operations as follows: the previous row of FM data is divided into 8 equal blocks to obtain 8 parallel points, and the data of the same channel group at the same position in each block is multiplied by the corresponding channel of the convolution kernel.

[0107] For example, FM uses the HWC88 format for depthwise convolution operations as follows: the 8 parallel points on FM are multiplied with the corresponding channels of the convolution kernel, the convolution kernel is copied into 8 copies and stored in the Block Mem storage area for computation, and the data format before and after copying is the HWC8 format.

[0108] In other words, the HWC88 format disclosed in this disclosure is a new data storage format defined independently based on the HWC format.

[0109] Definition 1: Select 8 points with different W values ​​in the same row of the feature map. Each point has 8 channels, stored in one memory unit (64 bytes), and C and W are both multiples of 8. This explains the meaning of the last two "8"s in HWC88 compared to the HWC format, and provides one definition of the HWC format.

[0110] From another perspective, for FM data, the HWC88 format divides a line or a portion of a line of FM data into 8 equal blocks. Each block contains wi (wi = W / 8) data points. Then, the data of the same channel group (8 channels per group) at the same position in each block, a total of ci groups (ci = C / 8), are placed at the same memory address. Priority is given to storing all channels of the same pixel in each block.

[0111] The storage format of HWC88 format FM data is as follows: Figure 8 As shown.

[0112] An example of HWC88: Figure 9 As shown, in this example, W=48 and C=16. After splitting in the W and C directions, each block of data contains 6 numbers in the W direction (wi=6) and 2 groups in the C direction (ci=2).

[0113] The advantage of the HWC88 data storage format is that it has fewer restrictions on the C value compared to the HWC format, saving data storage space in most cases where padding with zeros is required. For example, for an RGB image with C=3, if W=60 and H=1, storing it in the HWC format requires 60 memory units; while storing it in the HWC88 format only requires 8 memory units.

[0114] Furthermore, the HWC88 format disclosed in this disclosure can also be defined as follows:

[0115] Definition 2: For FM data, firstly, the entire input data fm is divided into segments, such as... Figure 10 As shown, the data between the two baffles in the figure represents a segment of data to be processed. Then, following the method described in Definition 1, in the W direction, this segment of data fm is divided into 8 equal blocks, each containing wi (wi = W / 8). In the C direction, it is divided into ci groups based on 8 (ci = C / 8). Data from the same position and channel group in each block is placed in the same memory unit, prioritizing the storage of all channels of pixels at the same position in each block. In the example figure, Wp = 48, C = 8, and the values ​​are variable. According to the requirements of the HWC88 format, both Wp and C are multiples of 8.

[0116] To distinguish them, the HWC88 format described in Definition 1 is called the basic HWC88 format, and the HWC88 format described in Definition 2 is called the improved HWC88 format.

[0117] like Figure 11 As shown, in this example, W=144, C=16, after being divided into 3 segments, each segment has Wp=48. After being divided in the W and C directions, each block of data contains 6 numbers in the W direction (wi=6) and 2 groups in the C direction (ci=2).

[0118] The advantage of the improved HWC88 format is that once the algorithm network is determined, the limitation on the image or feature map size is smaller. Taking a feature map with W=712 as an example, if stored in the basic HWC88 format, the wi of each block is calculated (wi=W / 8=712 / 8=89), and then data from the same position and channel group in each block are placed in the same memory unit. However, this block wi may not be suitable for the convolution operation of subsequent layers of the network, which will introduce additional data storage format conversion requirements. With the improved HWC88 format, the Wp of each segment and the corresponding wi of the blocks within the segment can be determined from the algorithm before the blocks are divided. For example, if Wp=128 (corresponding to wi=128 / 8=16), then the segments are divided according to Wp=128 for each segment. The first 5 segments are complete segments, and the last segment has only 72 valid points. The missing parts are padded with 0, and then the data within the segment is divided into blocks, and data from the same position and channel group in each block are placed in the same memory unit. The key point is that the HWC88 format, regardless of its definition, is used to implement the rearrangement of the format of FM in data storage, and needs to meet the following conditions: the format has good locality, can improve the utilization of data in the cache, and improve data transmission bandwidth and computing performance.

[0119] Furthermore, the HWC88 storage format can also be defined as follows:

[0120] If multiple images are processed in parallel during convolution, first take 8 images to be processed in parallel, and divide each row of the input data fm in the same row of each image into segments, such as... Figure 11AAs shown, the data between the two gray baffles represents a segment of data to be processed. Each image is then divided into ci groups (ci = C / 8) along the C direction with a base of 8. Data from the same channel group at the same position in each image is placed in the same memory location, prioritizing the storage of all channels for pixels at the same position within each block. In the part_M-1 and part_M+1 portions of the image, all channels for points with the same n and w values ​​(as shown in the dashed box, all channels with n = 7 and w = 47) have been merged into a simplified representation. This HWC88 format is called the multi-batch HWC88 format, and its advantage is that there are no restrictions on the W direction of fm, as long as C is a multiple of 8.

[0121] Furthermore, it can be observed that the basic HWC88 format is a special case of the improved HWC88 format, where a single line of data in the fm is divided into one segment, and Wp = W. The improved HWC88 format is an extension of the basic HWC88 format, especially for larger FM sizes.

[0122] In another embodiment, 8 channels are grouped together.

[0123] In another embodiment, the FM uses the HWC88 format for convolution operations as follows: the previous row of FM data is divided into 8 equal blocks to obtain 8 parallel points, and the data of the same channel group at the same position in each block is multiplied with the corresponding channel of the Kernel.

[0124] In this embodiment, Weight data (i.e., WT data) is stored in memory in HWC88 format, but the format of Weight may vary depending on the FM data format used during calculation and the type of calculation. Typically, WT data is formatted offline.

[0125] If FM uses the HWC88 format for convolution operations, the parallelism in the CH direction is 8, meaning 8 channels are grouped together. Simultaneously, the parallelism in the N direction is also 8, meaning 8 kernels are grouped together. WT stores the 8 channels of a corresponding group of kernels at a single address, such as... Figure 12 As shown, first store the 4 channels of the P0 point (H=0, W=0) of the first group of 8 kernels, until all 9 points of the first group of 8 kernels are stored, then start storing the data of the second group of 8 kernels.

[0126] Furthermore, in one embodiment, the neural network preprocessing method disclosed herein can be implemented as follows: Figure 13 The hardware implementation shown. Based on the above Figure 13 The illustrated hardware architecture can be used as a reference. Figure 14 When rearranging, a specific data processing is performed, in which:

[0127] The entire reorder process is mainly divided into two parts: one is to rearrange the channels of the input image data, and the other is to convert the format of the rearranged data. The format conversion part is implemented by instantiating 8 FIFOs. It should be noted that the input and output bit widths of a FIFO are generally 8, with the output bit width fixed at 8. The maximum input bit width can be the number of input channels after image rearrangement. The input bit width must meet two basic conditions: first, it must be an integer multiple of 8; second, it must be divisible by the number of input channels after image rearrangement. Taking a rearranged image with 32 input channels as an example, the FIFO input bit width can only be 8, 16, or 32. The larger the input bit width, the faster the reorder processing speed.

[0128] Although the embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this disclosure is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of this disclosure, and all of these are within the scope of protection of this disclosure.

Claims

1. A neural network preprocessing method, characterized in that: If each convolution requires m points and n channels of data, then during rearrangement: In the first column, first arrange the m*n numbers involved from the first point to the mth point from bottom to top; Furthermore, if the convolution stride is k, then when rearranging from left to right to the second column: the data of the n channels of the (k+1)th point are first arranged from the bottom; then, the (k+2)th point is arranged sequentially from bottom to top, all the way up to the (k+m)th point; Furthermore, when rearranging to the (i+1)th column, the data from the n channels of the i*k+1, i*k+2, …, i*k+m points are arranged from bottom to top. If N data is needed for each convolution, N corresponds to the computational parallelism of the neural network accelerator. When N is greater than m*n channels of data, then from the first column to the second to last column, each column can be padded with N-(m*n) zeros. For the last column, add NM zeros based on the actual number M of channel data in the last column; The rearrangement process follows a conversion method for HWC data format, including the following steps: S1: Read image data from DDR main memory. The original image data is stored in main memory in HWC format by default. The image is 256*256, so in each cycle, 16 points of 4-channel data values ​​are read from DDR. S2: The read data is processed by rearrangement and padding with zeros to achieve channel rearrangement and channel compensation, where 0*4 represents the 4 compensated channels 0, and the width of each line of data after compensation is 128. S3: Store the rearranged data into 8 FIFO first-in-first-out memories. The input width of each FIFO is 16 bytes and the output width is 8 bytes. Each FIFO stores 128 / 8=16 data points. The first FIFO stores the first 16 columns corresponding to points 0-15, and the last FIFO stores the last 16 columns of data. S4: Previously, the 8 FIFOs were written sequentially. This step reads them in parallel, that is, reads the data from all 8 FIFOs simultaneously. For the first FIFO: since the output bit width is only 8 bytes, the 16 bytes of data written in the previous cycle will be read out in two cycles. The first cycle reads BGRX (0-1), and the second cycle reads BGRX2 and 4 0s. For the second FIFO, the data read in the first cycle is GBRX (32-33), and the data read in the first cycle of the eighth FIFO is GBRX (224-225). S5: The 8-byte data output from the 8 FIFOs is concatenated into 64 bytes and written back to the DDR main memory. When all the data in the 8 FIFOs is written out, the format conversion of the image from HWC to HWC88 after channel rearrangement is completed. Thus, when all 256 lines of input image data have completed the five steps from S1 to S5, the entire original 4-channel 256*256 pixel image has also completed all rearrangement processing. The HWC88 storage format is specifically as follows: Divide a row or a portion of a row of FM data into 8 blocks. Place the data of the same channel group at the same position in each block at the same memory address, prioritizing the storage of all channels of the same pixel in each block.

2. The processing method according to claim 1, wherein, The HWC data format conversion method is as follows during rearrangement, including the following steps: S100: Each time, a whole line of data is read from the DDR main memory in the vertical direction. This is the data for one format conversion. S200: Store all data sequentially into 8 storage areas. Each Block Mem storage area stores CH*W / 8 data items, where CH represents the number of channels and W represents the width. S300: Reads data from 8 Block Mems in parallel, with each Block Mem outputting 8 bytes of data. The data is then concatenated into 64 bytes and stored back into DDR main memory.

3. The processing method according to claim 1 or 2, wherein, The HWC88 storage format is used to implement FM's reordering of data storage organization format, and it needs to meet the following conditions: the format has good locality, can improve the utilization of data in the cache, and improve data transmission bandwidth and computing performance.

4. The processing method according to claim 3, wherein, FM uses the HWC88 format for convolution operations, specifically by dividing the previous row of FM data into 8 equal blocks to obtain 8 parallel points, and multiplying the data of the same position and channel group in each block with the corresponding channel of the convolution kernel.

Images