Neural network data storage format conversion method suitable for general-purpose hardware circuits

By converting image data to HWC88 format, the problem of low computational efficiency of neural network models on edge platforms with limited memory resources and low processor performance is solved, thereby improving data transmission bandwidth and computational performance. This method is suitable for neural network data storage format conversion in general-purpose hardware circuits.

CN115373598BActive Publication Date: 2026-11-17XI AN JIAOTONG UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210923740.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2022-01-30
Filing Date
2022-08-02
Publication Date
2026-11-17
Estimated Expiration
2042-08-02

AI Technical Summary

Technical Problem

Existing neural network models are difficult to deploy and run in real time on edge platforms with limited memory resources and low processor performance, mainly due to the low computational efficiency caused by the difference in data storage formats on different hardware platforms.

Method used

Image data is converted using the HWC88 storage format. By transposing the image data from the source storage format to the HWC88 format and reading and writing it in the destination memory, the utilization rate of data in the cache and the data transmission bandwidth are improved. This method is suitable for converting neural network data storage formats in general-purpose hardware circuits.

Benefits of technology

It improves data transmission bandwidth and computing performance, enhances the inference efficiency of neural networks on edge platforms, and is suitable for hardware environments with limited memory resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115373598B_ABST
    Figure CN115373598B_ABST
Patent Text Reader

Abstract

This disclosure discloses a computer-implemented data processing method, including the following steps: S100: reading image data from a source memory, wherein the source memory has a source storage format, and the reading from the source memory is performed in a mode suitable for the source memory; S200: transposing the image data from the source storage format to a destination storage format different from the source storage format, wherein one of the source storage format and the destination storage format is an HWC8 storage format, and the other of the source storage format and the destination storage format is an HWC88 storage format; S300: writing the image data to the destination memory, wherein the destination memory has a destination storage format, and the reading from the destination memory is performed in a format suitable for the destination memory. This disclosure utilizes the better locality of the HWC88 storage format, which can improve the utilization rate of data in the cache, thereby improving data transmission bandwidth and computing performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure belongs to the field of artificial intelligence technology, and specifically relates to a method for converting neural network data storage formats suitable for general-purpose hardware circuits. Background Technology

[0002] In recent years, with the development of artificial intelligence technology, primarily based on convolutional neural networks, more and more computer vision tasks have been well solved, such as image classification, object detection, and semantic segmentation. A current trend is to deploy high-performance neural network models on edge platforms and enable them to run in real-time (more than 30 frames per second) in real-world scenarios, such as mobile / embedded devices. However, these platforms are characterized by limited memory resources, low processor performance, and power consumption. This makes it impossible to deploy even the most accurate models on these platforms and achieve real-time performance due to their excessive demands on memory and computing resources.

[0003] In deep learning, data formats such as NCHW, NHWC, and CHWN are used to improve data transfer bandwidth and computational performance. These are logical data formats for images or feature maps (the order in which data is stored in memory). Different data formats can increase the reusability of local data and improve the utilization of data in the cache. Using different data storage formats on different hardware platforms directly affects computational efficiency. For example, the default data format in TensorFlow is NHWC because it is slightly faster than NCHW on CPU platforms, while NCHW is the default format for Nvidia cuDNN, and it is faster when using GPU acceleration. Summary of the Invention

[0004] In view of this, this disclosure provides a computer-implemented data processing method, including:

[0005] S100: Read image data from a source memory, wherein the source memory has a source storage format, and the reading from the source memory is performed in a mode suitable for the source memory;

[0006] S200: Transpose the image data from the source storage format to a destination storage format different from the source storage format, wherein one of the source storage format and the destination storage format is HWC8 storage format, and the other of the source storage format and the destination storage format is HWC88 storage format;

[0007] S300: Write image data to a destination memory, wherein the destination memory has the destination storage format, and the reading of the destination memory is performed in a format suitable for the destination memory.

[0008] Preferred,

[0009] The HWC88 storage format is used to implement FM's reordering of data storage organization format, and it needs to meet the following conditions: the format has good locality, can improve the utilization of data in the cache, and improve data transmission bandwidth and computing performance.

[0010] Preferred,

[0011] The image data is neural network data.

[0012] Preferred,

[0013] The image data includes FM data, WT data, Bias data, and Clip data.

[0014] Preferred,

[0015] Step S200 further includes the following steps:

[0016] S201: Load an entire row of data in direction H from the double-rate synchronous dynamic random access memory (DDR) each time;

[0017] S202: The data is sequentially stored into 8 storage areas Block Mem, each storage area Block Mem stores CH*W / 8 data, where CH represents the channel and W represents the width;

[0018] S203: Read the data in the eight storage areas Block Mem in parallel. Each storage area Block Mem has an output bit width of 8B, and store the data in Double Rate Synchronous Dynamic Random Memory (DDR).

[0019] Preferred,

[0020] The HWC88 storage format is specifically as follows:

[0021] Divide a row or a portion of a row of FM data into 8 blocks. Place the data of the same channel group at the same position in each block at the same memory address, prioritizing the storage of all channels of the same pixel in each block.

[0022] Preferred,

[0023] Eight channels form a group.

[0024] Preferred,

[0025] FM uses the HWC88 format for convolution operations, specifically by dividing the previous row of FM data into 8 equal blocks to obtain 8 parallel points, and multiplying the data of the same position and channel group in each block with the corresponding channel of the convolution kernel.

[0026] Preferred,

[0027] FM uses the HWC88 format for depthwise convolution operations. Specifically, the 8 parallel points on FM are multiplied by the corresponding channels of the convolution kernel. The convolution kernel is copied into 8 copies and stored in the Block Mem storage area for computation. The data format before and after copying is HWC8 format.

[0028] It is understood that the above solution actually discloses a method for converting neural network data storage formats suitable for general-purpose hardware circuits. Through the above technical solution, the HWC88 storage format used in this disclosure has better locality, which can improve the utilization rate of data in the cache, thereby improving data transmission bandwidth and computing performance. This solution is of great significance for solving the inference efficiency problem in artificial intelligence applications. The change in the storage format of the neural network data in memory can realize the rearrangement of the FM organization format in data storage. Attached Figure Description

[0029] Figure 1 This is a flowchart of a computer-implemented data processing method provided in one embodiment of the present disclosure;

[0030] Figure 2 This is a diagram illustrating the storage formats of HWC8, HWC32, and HWC64 FM data in one embodiment of this disclosure.

[0031] Figure 3 This is a diagram illustrating the storage format of HWC88 format FM data in one embodiment of this disclosure;

[0032] Figure 4 This is an example diagram of a basic HWC88 in one embodiment of this disclosure;

[0033] Figure 5 This is an example diagram of segmenting a single line of the entire input data fm in one embodiment of this disclosure;

[0034] Figure 6 This is an example diagram of an improved HWC88 in one embodiment of this disclosure;

[0035] Figure 6A This is a schematic diagram of the HWC88 storage format for multiple batches in one embodiment of this disclosure;

[0036] Figure 7 This is a diagram showing the storage format of WT data when FM performs convolution operations using the HWC88 format in one embodiment of this disclosure;

[0037] Figure 8This is a schematic diagram of the convolution of a three-channel color input image in one embodiment of this disclosure;

[0038] Figure 9 This is a diagram showing that FM performs convolution operations in HWC format in one embodiment of this disclosure;

[0039] Figure 10 This is a diagram of FM performing convolution operations in HWC88 format in one embodiment of this disclosure;

[0040] Figure 11 This is a graph of depthwise convolution operation performed by FM in one embodiment of this disclosure;

[0041] Figure 12 This is the storage format of WT data when FM performs depthwise convolution operations in HWC format in one embodiment of this disclosure;

[0042] Figure 13 This is a diagram showing the storage format of WT data when FM performs depthwise convolution operations using the HWC88 format in one embodiment of this disclosure;

[0043] Figure 14 This is a diagram illustrating the storage format of BC data in one embodiment of this disclosure;

[0044] Figure 15 This is a diagram illustrating the storage format of INT4, INT8, and INT16 data in one embodiment of this disclosure;

[0045] Figure 16 This is a diagram illustrating the mutual conversion between HWC8 format and HWC88 format data in one embodiment of this disclosure;

[0046] Figure 17 This is a diagram illustrating the format conversion of FM data via Block Mem in one embodiment of this disclosure;

[0047] Figure 18 This is a diagram illustrating the format conversion of general data in one embodiment of this disclosure. Detailed Implementation

[0048] See Figure 1 In one embodiment, a computer-implemented data processing method is disclosed, comprising:

[0049] S100: Read image data from a source memory, wherein the source memory has a source storage format, and the reading from the source memory is performed in a mode suitable for the source memory;

[0050] S200: Transpose the image data from the source storage format to a destination storage format different from the source storage format, wherein one of the source storage format and the destination storage format is HWC8 storage format, and the other of the source storage format and the destination storage format is HWC88 storage format;

[0051] S300: Write image data to a destination memory, wherein the destination memory has the destination storage format, and the reading of the destination memory is performed in a format suitable for the destination memory.

[0052] In this embodiment, the NPU needs to read FM, WT, Bias, Clip, or data used for general matrix multiplication (GEMM) during computation. To improve data transmission bandwidth and computational performance, this data is typically stored in memory in multiple logical formats (the order in which the data is stored in memory), such as HWC and CHW. The naming of HWC and CHW is based on the order of data arrangement across its dimensions (Height, Width, Channel). FM data can be divided into two categories, HWC and CHW, arranged according to their right-to-left priority. "C" represents the number of channels in the feature map, "H" represents the height of the feature map, and "W" represents the width of the feature map.

[0053] Although FM can be artificially divided into multiple dimensions, for computers, data storage can only be linear. The HWC and CHW formats mentioned above specify the order of linear data storage. Taking an RGB image as an example, in the HWC format, the C direction is arranged first, meaning that the R, G, and B channel data of one pixel are stored before the R, G, and B channel data of the next pixel are stored; while in the CHW format, the C direction is arranged last, meaning that the R channel data of all pixels in the entire image are stored first, then the G channel data, and finally the B channel data.

[0054] Standard FM data formats typically include HWC, HWC8, HWC32, and HWC64. The latter three are based on the standard HWC format but add restrictions on the number of channels. HWC8 indicates a multiple of 8 channels; HWC32 indicates a multiple of 32 channels; and HWC64 indicates a multiple of 64 channels. For FM channels less than 8, 32, or 64, zeros should be added for alignment. Their memory storage format is as follows: Figure 2 As shown.

[0055] The definitions of terms appearing in this disclosure are shown in Table 1.

[0056]

[0057] Table 1

[0058] In another embodiment, the image data is neural network data.

[0059] In another embodiment, the image data includes FM data, WT data, Bias data, and Clip data.

[0060] In another embodiment, the HWC88 storage format is specifically as follows:

[0061] Divide a row or a portion of a row of FM data into 8 blocks. Place the data of the same channel group at the same position in each block at the same memory address, prioritizing the storage of all channels of the same pixel in each block.

[0062] In this embodiment, a row or a portion of a row is divided. This means that a whole row of FM data can be divided into any number of parts first, and then each part is organized and transformed. Each part of FM data is divided into 8 blocks on average. For FM data that cannot be divided equally, zeros can be added so that its width direction and CH channel direction are integer multiples of 8. Data of the same channel group (8 channels as a group) at the same position in each block are placed at the same address in memory. Priority is given to storing all channels of the same position of the pixels in each block in HWC88 format.

[0063] The HWC88 format disclosed in this disclosure is a new data storage format defined independently based on the HWC format.

[0064] Definition 1: Select 8 points with different W values ​​in the same row of the feature map. Each point has 8 channels, stored in one memory unit (64 bytes), and C and W are both multiples of 8. This explains the meaning of the last two "8"s in HWC88 compared to the HWC format, and provides one definition of the HWC format.

[0065] From another perspective, for FM data, the HWC88 format divides a row or a portion of a row of FM data into eight equal blocks. Each block contains wi (wi = W / 8) data points. Then, within each block, the data from the same channel group (8 channels per group), totaling ci groups (ci = C / 8), are stored at the same memory address, prioritizing the storage of all channels for pixels at the same location within each block. The storage format of HWC88 FM data is as follows: Figure 3 As shown.

[0066] An example of HWC88: Figure 4 As shown, in this example, W=48 and C=16. After splitting in the W and C directions, each block of data contains 6 numbers in the W direction (wi=6) and 2 groups in the C direction (ci=2).

[0067] The advantage of the HWC88 data storage format is that it has fewer restrictions on the C value compared to the HWC format, saving data storage space in most cases where padding with zeros is required. For example, for an RGB image with C=3, if W=60 and H=1, storing it in the HWC format requires 60 memory units; while storing it in the HWC88 format only requires 8 memory units.

[0068] Furthermore, the HWC88 format disclosed in this disclosure can also be defined as follows:

[0069] Definition 2: For FM data, firstly, the entire input data fm is divided into segments, such as... Figure 5 As shown, the data between the two baffles in the figure represents a segment of data to be processed. Then, following the method described in Definition 1, in the W direction, this segment of data fm is divided into 8 equal blocks, each containing wi (wi = W / 8). In the C direction, it is divided into ci groups based on 8 (ci = C / 8). Data from the same position and channel group in each block is placed in the same memory unit, prioritizing the storage of all channels of pixels at the same position in each block. In the example figure, Wp = 48, C = 8, and the values ​​are variable. According to the requirements of the HWC88 format, both Wp and C are multiples of 8.

[0070] To distinguish them, the HWC88 format described in Definition 1 is called the basic HWC88 format, and the HWC88 format described in Definition 2 is called the improved HWC88 format.

[0071] like Figure 6 As shown, in this example, W = 144, C = 16, and after being divided into 3 segments, each segment has Wp = 48. After being divided in the W and C directions, each block of data contains 6 numbers in the W direction (wi = 6) and 2 groups in the C direction (ci = 2).

[0072] The advantage of the improved HWC88 format is that once the algorithm network is determined, the limitation on the image or feature map size is smaller. Taking a feature map with W=712 as an example, if stored in the basic HWC88 format, the wi of each block is calculated (wi=W / 8=712 / 8=89), and then data from the same position and channel group in each block are placed in the same memory unit. However, this block wi may not be suitable for the convolution operation of subsequent layers of the network, which will introduce additional data storage format conversion requirements. With the improved HWC88 format, the Wp of each segment and the corresponding wi of the blocks within the segment can be determined from the algorithm before the blocks are divided. For example, if Wp=128 (corresponding to wi=128 / 8=16), then the segments are divided according to Wp=128 for each segment. The first 5 segments are complete segments, and the last segment has only 72 valid points. The missing parts are padded with 0, and then the data within the segment is divided into blocks, and data from the same position and channel group in each block are placed in the same memory unit. The key point is that the HWC88 format, regardless of its definition, is used to implement the rearrangement of the format of FM in data storage, and needs to meet the following conditions: the format has good locality, can improve the utilization of data in the cache, and improve data transmission bandwidth and computing performance.

[0073] Furthermore, the HWC88 storage format can also be defined as follows:

[0074] If multiple images are processed in parallel during convolution, first take 8 images to be processed in parallel, and divide each row of the input data fm in the same row of each image into segments, such as... Figure 6A As shown, the data between the two gray baffles represents a segment of data to be processed. Each image is then divided into ci groups (ci = C / 8) along the C direction with a base of 8. Data from the same channel group at the same position in each image is placed in the same memory location, prioritizing the storage of all channels for pixels at the same position within each block. In the part_M-1 and part_M+1 portions of the image, all channels for points with the same n and w values ​​(as shown in the dashed box, all channels with n = 7 and w = 47) have been merged into a simplified representation. This HWC88 format is called the multi-batch HWC88 format, and its advantage is that there are no restrictions on the W direction of fm, as long as C is a multiple of 8.

[0075] Furthermore, it can be observed that the basic HWC88 format is a special case of the improved HWC88 format, where a single line of data in the fm is divided into one segment, and Wp = W. The improved HWC88 format is an extension of the basic HWC88 format, especially for larger FM sizes. Since the example FM sizes discussed later are relatively small, the discussion will consistently follow the basic HWC88 format.

[0076] In another embodiment, 8 channels are grouped together.

[0077] In another embodiment, the FM uses the HWC88 format for convolution operations as follows: the previous row of FM data is divided into 8 equal blocks to obtain 8 parallel points, and the data of the same channel group at the same position in each block is multiplied by the corresponding channel of the Kernel.

[0078] In this embodiment, Weight data (i.e., WT data) is stored in memory in HWC88 format, but the format of Weight may vary depending on the FM data format used during calculation and the type of calculation. Typically, WT data is formatted offline.

[0079] If FM uses the HWC88 format for convolution operations, the parallelism in the CH direction is 8, meaning 8 channels are grouped together. Simultaneously, the parallelism in the N direction is also 8, meaning 8 kernels are grouped together. WT stores the 8 channels of a corresponding group of kernels at a single address, such as... Figure 7 As shown, first store the 4 channels of the P0 point (H=0, W=0) of the first group of 8 kernels, until all 9 points of the first group of 8 kernels are stored, then start storing the data of the second group of 8 kernels.

[0080] If FM uses the HWC format for traditional convolution operations, the detailed calculation process is as follows.

[0081] Fixed-point convolution:

[0082] like Figure 8 As shown, for a three-channel color input image, after passing through a convolutional layer with a 3×3 kernel (assuming the number of output channels is 4, then the kernel shape is 3×3×3×4), 4 feature maps are finally output.

[0083] For a typical input FM:

[0084]

[0085] Where Ofm is the output Feature Map, ifm is the input Feature Map, k is the kernel convolution kernel, clip is the re-quantization operation, which means that the re-quantization parameters used belong to the integer set, b is the bias offset, and Z represents the integer set.

[0086] Depend on Figure 9 The basic steps of traditional convolution can be seen as follows:

[0087] STEP 1: First, perform the inner product multiplication of the matrices to find... Figure 9 The inner product of the two dark matrices in the input fm and kernel is obtained by multiplying the elements at the same positions of the two matrices to get a new matrix.

[0088] STEP 2: Next, perform a full addition calculation. Add all the values ​​of the new matrix obtained in step 1 to get a new value. Then add a bias to this value to get the result of one convolution, which is also the result of one point on the output feature image.

[0089] STEP3: Repeat the STEP1 and STEP2 processes sequentially over the entire FM image to obtain the complete output feature image output fm.

[0090] As can be seen from the above steps, each output FM point is calculated by taking the inner product of all channels of the corresponding input FM point and the corresponding position of the kernel.

[0091] In other words, traditional convolution has the following data reuse relationships:

[0092] Different channels of the same output FM point reuse the same input FM data.

[0093] The same channel at different output FM points reuses the same kernel data.

[0094] The input FM data required by adjacent output FM points largely overlaps.

[0095] Because image data is stored sequentially in main memory, but the corresponding data blocks required for convolution operations are not contiguous in hardware storage, this results in a large number of data calls and address jumps during the convolution process. Furthermore, data scheduling typically introduces significant time overhead into the processor. This demonstrates that the traditional HWC data storage format leads to low data utilization in the cache during each call, and the storage format lacks good locality.

[0096] Based on in-depth analysis of data reuse relationships, we propose a convolution operation in the HWC88 format, such as... Figure 10 As shown, the convolution operation method based on the HWC88 format includes the following steps:

[0097] STEP 1: First, perform the inner product of the matrices as before. Figure 10 The dark parts in the input fm and kernel are concatenated into two 8*8 matrices and their inner product is calculated. That is, the elements at the same position in the two matrices are multiplied to obtain a new matrix.

[0098] STEP2: The new matrix obtained from STEP1 is the partial convolution result of the dark part in the output feature image, which we call the partial sum (Psum). By repeating the STEP1 process in a regular order throughout the entire FM image and kernel, we can obtain the complete partial sum (Psum) corresponding to each point of all output feature maps.

[0099] STEP 3: Finally, perform a full addition calculation. Add all the corresponding values ​​of the parts obtained in STEP 2 (Psum) together to get a new value. Then add the bias to this value to get the result of one convolution, which is also the result of one point on the output feature image.

[0100] It's important to note that, taking a 3x3 convolution as an example, traversing a complete FM (Factor-Minimum) yields a complete partial sum (Psum). Traversing a complete 3x3 kernel provides nine partial sums (Psum) for a given point. Performing a full addition on these nine partial sums, plus the bias, gives the result for that point in the output feature image. In HWC88 format convolutions, the traversal of both the FM and kernel dimensions is performed simultaneously. Therefore, the full addition can be performed when all partial sums for any point in the output feature image are obtained, without waiting for the complete traversal of all partial sums.

[0101] If FM uses the HWC88 format for convolution operations, the previous row of FM data is divided into 8 equal blocks to obtain 8 parallel points. Data from the same position and channel group (8 channels per group) in each block is multiplied by the corresponding channel in the kernel. This transforms the traditional convolution operation into a batch matrix-matrix operation. Simultaneously, all consecutive data retrieved from main memory and cache can be directly computed, significantly increasing data utilization. The main memory call logic is also simplified to sequential calls, reducing logical branches and address jumps, making it more hardware-friendly.

[0102] In another embodiment, the depthwise convolution operation of FM using HWC88 format is specifically as follows: the 8 parallel points on FM are multiplied with the corresponding channels of the Kernel, the kernel is copied into 8 copies and stored in Block Mem for computation, and the data format before and after copying is HWC8 format.

[0103] In this embodiment, unlike conventional convolution operations, each kernel of a depthwise convolution is responsible for one channel, and each channel is convolved by only one kernel. The convolution mentioned above operates on each channel of the input image simultaneously.

[0104] For a given three-channel color input image, Depthwise convolution first performs a single convolution operation, which is entirely within a two-dimensional plane. The number of convolution kernels is the same as the number of channels in the previous layer (one channel corresponds to one convolution kernel). Therefore, a three-channel image generates three feature maps after processing, as shown below. Figure 11 As shown, the depthwise convolution operation method includes the following steps:

[0105] S1: First, perform the inner product multiplication of the matrices to find... Figure 11 The inner product of two dark gray matrices in the input fm and kernel is obtained by multiplying the elements at the same position in the two matrices to get a new matrix.

[0106] S2: Next, perform full addition. Add all the values ​​of each channel of the new matrix obtained in S1 separately. This will result in new values ​​in different channels. Then, add a bias to these values ​​to obtain the result of one depthwise convolution, which is also the result of each channel of a point in the output feature image.

[0107] S3: On the entire FM image, repeat the process of S1 and S2 in a regular order to finally obtain the complete output feature image output fm.

[0108] If FM uses the HWC format for depthwise convolution operations, such as Figure 12 As shown, each address can store data for 64 channels. The storage order in the N, H, and W directions is the same as in the HWC88 mode.

[0109] If FM uses the HWC88 format for depthwise convolution operations, such as Figure 13 As shown, the 8 parallel points on the FM will be multiplied by the corresponding channel of the Kernel. Therefore, the kernel needs to be copied into 8 copies and stored in Local Memory for calculation. The data format before and after copying is the standard format, that is, the number of channels is a multiple of 8.

[0110] In another embodiment, the HWC88 storage format for BC data is as follows: In the current design, each output channel has a set of Bias and Clip data, and its storage format is as follows: Figure 14 As shown. Figure 14It provides storage formats for BC data in two precisions: INT8 and INT16.

[0111] In another embodiment, such as Figure 15 As shown, MAC calculation only supports three precisions: INT4, INT8, and INT16. The storage format of INT4 and INT16 data is similar to that of INT8. The difference is that INT8 stores data from 8 channels in each small block (8B), INT4 stores data from 16 channels in each small block (8B), while INT16 occupies two consecutive memory addresses (8B*2) to store the lower 8 bits and higher 8 bits of data from 8 channels respectively.

[0112] In another embodiment, step S200 described above further includes the following steps:

[0113] S201: Load an entire row of data in direction H from the double-rate synchronous dynamic random access memory (DDR) each time;

[0114] S202: The data is sequentially stored into 8 storage areas Block Mem, each storage area Block Mem stores CH*W / 8 data, where CH represents the channel and W represents the width;

[0115] S203: Read the data in the eight storage areas Block Mem in parallel. Each storage area Block Mem has an output bit width of 8B, and store the data in Double Rate Synchronous Dynamic Random Memory (DDR).

[0116] In this embodiment, the data format conversion module is mainly used to convert various data formats, primarily the conversion between HWC8 and HWC88. The following... Figure 16 For example, the original FM is CH(40)*W(16), and the FM data is stored in the formats HWC8 and HWC88 respectively.

[0117] This article uses FM as an example to introduce the general conversion method between HWC8 and HWC88, including the following steps:

[0118] Step 1: Each time, load a complete line of data from the DDR Load H direction; this constitutes one format conversion. Figure 16 In the example, H=1 in the original FM, and at this time, one row of data is all the data in the figure where CH=40*W=16;

[0119] Step 2: Store all data sequentially into 8 Block Mems. Each Block Mem contains CH (number of channels) * W (width) / 8 data entries. Figure 17 As shown;

[0120] Step 3: Read the data from the 8 Block Mems in parallel. Each Block Mem outputs a bit width of 8B, and then concatenate the data into a 64B block and store it in the DDR.

[0121] In another embodiment, a unified and universal format conversion method is proposed:

[0122] The previous example mentioned a method for converting FM data formats. This article proposes a more unified and universal data format conversion method applicable to various data types, including WT, BC, and FM. The principle is as follows: Figure 18 The diagram shows column transformations and matrix transpose.

[0123] The method for general format conversion includes the following steps:

[0124] ST1: First, the data columns are transformed. The data requiring format conversion is grouped into groups of 64B*8. Here, 64B refers to a memory bit width granularity of 64B. For cases where the memory bit width is not 64B, 64B of data can be read from memory each time. The 8 indicates that at least 8 columns of 64B data are needed for a subsequent matrix transpose. The address interval between these 8 columns is consistent and can be set according to the total data volume. In this example, there are a total of 16*64B of data, so the address interval is 16 / 8 = 2. Therefore, columns 0, 2, 4…14 are grouped together through column transformation, and columns 1, 3, 5…15 are grouped into a second group, as shown below. Figure 18 As indicated by the arrow;

[0125] ST2: Next is the matrix transpose. Each matrix refers to a set of 64B*8 data blocks that were transformed from ST1 through column transformations, such as... Figure 18 As shown in the dashed box. It's important to note that this is an 8x8 matrix, and each element has a unit of 8 bytes.

[0126] Although the embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this disclosure is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of this disclosure, and all of these are within the scope of protection of this disclosure.

Claims

1. A computer-implemented data processing method, comprising: S100: Read image data from a source memory, wherein the source memory has a source storage format, and the reading from the source memory is performed in a mode suitable for the source memory; S200: Transpose the image data from the source storage format to a destination storage format different from the source storage format, wherein one of the source storage format and the destination storage format is HWC8 storage format, and the other of the source storage format and the destination storage format is HWC88 storage format; S300: Write image data to a destination memory, wherein the destination memory has the destination storage format, and reads from the destination memory are performed in a format suitable for the destination memory; Step S200 further includes the following steps: S201: Load an entire row of data in direction H from the double-rate synchronous dynamic random access memory (DDR) each time; S202: The data is sequentially stored into 8 storage areas Block Mem, each storage area Block Mem stores CH*W / 8 data, where CH represents the channel and W represents the width; S203: Read the data in the eight storage areas Block Mem in parallel. Each storage area Block Mem has an output bit width of 8B, and store the data in Double Rate Synchronous Dynamic Random Memory (DDR).

2. The method according to claim 1, wherein, The HWC88 storage format is used to implement FM's reordering of data storage organization format, and it needs to meet the following conditions: the format has good locality, can improve the utilization of data in the cache, and improve data transmission bandwidth and computing performance.

3. The method according to claim 1, wherein, The image data is neural network data.

4. The method according to claim 3, wherein, The image data includes feature map (FM) data, weight (WT) data, bias (Bias) data, and requantization operation parameter (Clip) data.

5. The method according to claim 1, wherein, The HWC88 storage format is specifically as follows: Divide a row or a portion of a row of FM data into 8 blocks. Place the data of the same channel group at the same position in each block at the same memory address, prioritizing the storage of all channels of the same pixel in each block.

6. The method according to claim 5, wherein, Eight channels form a group.

7. The method according to claim 6, wherein, FM uses the HWC88 format for convolution operations, specifically by dividing the previous row of FM data into 8 equal blocks to obtain 8 parallel points, and multiplying the data of the same position and channel group in each block with the corresponding channel of the convolution kernel.

8. The method according to claim 6, wherein, FM uses the HWC88 format for depthwise convolution operations. Specifically, the 8 parallel points on FM are multiplied by the corresponding channels of the convolution kernel. The convolution kernel is copied into 8 copies and stored in the Block Mem storage area for computation. The data format before and after copying is HWC8 format.

Citation Information

Patent Citations

  • ZYNQ FPGA chip, data processing method thereof and storage medium

    CN110309088A

  • Extensible multi-precision data pipeline for computing non-linear and arithmetic functions in artificial neural networks

    US20210349718A1