A method for hardware automatic padding of convolution operation

Abstract

The present disclosure discloses a method of hardware automatic padding of convolution operation and a data storage format thereof. In the data storage process, the entire input data ifm is segmented, each segment of data is divided into 8 equal parts to obtain 8 groups of data, and then each group of data is extracted at equal positions, the equal position data is arranged in a column, and sequential storage is performed. In this way, this storage format has better locality. Compared with the HWC format in the prior art, the present disclosure names this storage format as HWC88 format. For the method of hardware automatic padding, padding data is automatically supplemented by copying and shifting the original data on the left and right sides of the data stored in the HWC88 format, so that the convolution operation is quickly performed, and when the padding is explicitly stored, the padding part is regarded as normal data, and no additional padding address calculation is required.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the field of processor and computing technology, and particularly relates to a method for hardware automatic padding of convolution operation and a data storage format thereof. BACKGROUND

[0002] In recent years, in order to improve the data transmission bandwidth and computing performance, image or feature map uses NCHW, NHWC and other logical data formats (data storage order in memory) for data storage. In the convolutional neural network, in order to make the output feature map (ofm) have the same width w and height h as the input feature map (ifm) after the convolution operation, padding operation is often involved. In different convolution operations, the padding position and size of the above-mentioned data format are dynamically changed, which leads to the need for additional calculation of padding address when padding is performed during convolution operation. SUMMARY

[0003] In view of this, the present disclosure provides a method for hardware automatic padding of convolution operation, characterized in that:

[0004] The method uses an HWC88 data storage format to store data, wherein,

[0005] In the data storage process, the entire input data ifm is segmented, each segment of data is divided into 8 equal parts to obtain 8 groups of data, and then each group of data is extracted at an equal position. The data at the equal position is arranged in a column and stored in sequence.

[0006] The method automatically fills padding data by copying and shifting the original data from the left and right sides of the data stored in the HWC88 format, so as to quickly perform convolution operation. When padding is explicitly stored, the padding part is regarded as normal data, and no additional padding address needs to be calculated.

[0007] Preferably,

[0008] For different wt sizes of convolution operation, in order to keep the size of ofm after convolution operation the same as ifm, when multiple rows of padding are needed for image or ifm data, the method fills padding data by copying and shifting the original data from the left and right sides of the fm storage data, and then filling the empty bits with the data column after segmenting the fm to fill padding data, i.e. the data on the left and right sides of the current segment, to fill padding data.

[0009] Preferably,

[0010] For different Channel numbers,

[0011] Suppose the dimension of fm is 2 (height) * 24 (width) * C (channel), and in the Channel direction, it is divided into 0~ci segments (ci = C / 8) based on 8. When storing fm in HWC88 format, the data of the same position and the same channel group in each block is placed in the same address of the memory, and the data of all channels of the same position in each block is stored preferentially.

[0012] Suppose the bit width of a column of memory is 64B, which can store the data of 8 points, and each point has 8 channels. The address offset between two adjacent points in the Width direction of fm is 64B*ci, which is recorded as block_offset.

[0013] When the padding is stored as implicit storage, the data of the source position is copied once. In Pa.b, a represents the bit order in the Width direction, and b represents the bit order in the Channel direction.

[0014] Preferably,

[0015] Suppose the dimension of fm is 1 (height) * 8 (width) * 8 (channel), and the dimension of wt is 5 (height) * 5 (width) * 8 (channel) * 8 (n). When storing fm in HWC88 format, ifm needs to be supplemented with 2 rows of padding.

[0016] For the case of only 1 column after storing in HWC88 format, first, according to the explicit storage method, the first row of padding of fm is explicitly supplemented to obtain a 3-column explicitly stored fm.

[0017] After copying and shifting the original data of the data source, the empty bits are supplemented with the data columns before and after the fm segmentation. It is easy to know that the data in the positions of [part_M-1_tail] and [part_M+1_head] are the real padding.

[0018] According to the implicit storage method, the above fm is supplemented with the second row of padding, and after copying and shifting the original data, the empty bits are supplemented with the data columns before and after the fm segmentation. Thus, data reading can be performed according to the implicit storage method.

[0019] Preferably,

[0020] The HWC88 storage format is used to realize rearrangement of the FM in the data storage organization format, and needs to meet the following conditions: the format has good locality, can improve the utilization of data in the cache, and improve the data transmission bandwidth and computing performance.

[0021] Preferably,

[0022] The HWC88 storage format is specifically:

[0023] The FM data in a row or a part of a row is evenly divided into 8 blocks, the data of the same position and the same channel group in each block is placed in the same address of the memory, and all channels of the same position in each block are preferentially stored.

[0024] Preferably,

[0025] The FM adopts the HWC88 format for convolution operation, specifically: the FM data in a row is evenly divided into 8 blocks to obtain 8 parallel points, and the data of the same position and the same channel group in each block is multiplied by the corresponding channel of the convolution kernel.

[0026] Therefore, the disclosure discloses a hardware automatic padding method for convolution operation and a data storage format thereof. The automatic padding scheme based on the HWC88 format automatically fills padding data by copying and shifting the original data when storing padding data implicitly, and fills padding data on the left and right sides of the data using the basic or improved HWC88 storage format disclosed in the disclosure, thereby quickly performing convolution calculation; in the HWC88 storage format using multiple batches, padding values are filled on the left and right sides of the data. When storing explicitly, the padding part is regarded as normal data, and no additional padding address calculation is needed.

[0027] In addition, the HWC88 format proposed in the disclosure needs to meet the following conditions: the format has good locality, can improve the utilization of data in the cache, and improve the data transmission bandwidth and computing performance. In this way, the storage format has better locality.

[0028] Therefore, compared with the HWC format in the prior art, the storage format is named as the HWC88 format in the present disclosure. For the hardware automatic padding method disclosed in the present disclosure, padding data is automatically supplemented on the left and right sides of data in the HWC88 storage format by copying and shifting the original data, so that the convolution operation is quickly performed, and when the padding is stored explicitly, the padding part is regarded as normal data, and the padding address does not need to be calculated additionally. It should be noted that, since the HWC88 data storage format has better locality, during the data storage process, the entire input data ifm is segmented, each segment of data is divided into 8 equal parts, 8 groups of data are obtained, and then each group of data is extracted at an equal position, the equal position data is arranged in a column, and is sequentially stored. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1A is a schematic diagram of a padding method of ifm in four rounds of white strips in an embodiment of the present disclosure;

[0030] Figure 1B is a schematic diagram of a padding method of ifm in two rows of white strips in four rounds of ifm in an embodiment of the present disclosure;

[0031] Figure 2 is a schematic diagram of segmentation of fm in the HWC88 storage format in an embodiment of the present disclosure;

[0032] Figure 3 is a schematic diagram of padding data on the left and right sides of fm storage data in an embodiment of the present disclosure;

[0033] Figure 4 is a schematic diagram of implicit storage of padding in the HWC88 format in an embodiment of the present disclosure;

[0034] Figure 5A 、 Figure 5B is a schematic diagram of explicit storage of padding in the HWC88 format in an embodiment of the present disclosure;

[0035] Figure 6 is a schematic diagram of semi-explicit storage of padding in the HWC88 format in the case (3) and (4) in an embodiment of the present disclosure;

[0036] Figure 7A 、 Figure 7B is a schematic diagram of padding of ifm in two rows in an embodiment of the present disclosure;

[0037] Figure 7C is a schematic diagram of padding of ifm in one row in an embodiment of the present disclosure;

[0038] Figure 8 is a schematic diagram of implicit storage of padding in HWC88 format in one embodiment of the present disclosure;

[0039] Figure 9 is a schematic diagram of first row padding of fm being filled according to the explicit storage method to obtain 3-column explicit storage of fm in one embodiment of the present disclosure;

[0040] Figure 10 is a schematic diagram of second row padding of the fm being filled according to the implicit storage method in one embodiment of the present disclosure;

[0041] Figure 11 is a schematic diagram of FM data being stored in HWC88 format in one embodiment of the present disclosure;

[0042] Figure 12 is a schematic diagram of a basic HWC88 storage in one embodiment of the present disclosure;

[0043] Figure 13 is a schematic diagram of one row of the input data fm being segmented in one embodiment of the present disclosure;

[0044] Figure 14 is a schematic diagram of an improved HWC88 in one embodiment of the present disclosure;

[0045] Figure 15 is a schematic diagram of WT data storage format when FM adopts HWC88 format for convolution operation in one embodiment of the present disclosure;

[0046] Figure 16 is a simplified schematic diagram of data storage format in the memory in one embodiment of the present disclosure;

[0047] Figure 17 is a schematic diagram of multi-batch HWC88 storage format in one embodiment of the present disclosure;

[0048] Figure 18 is a schematic diagram of padding supplement when HWC88 storage format is used for convolution operation in one embodiment of the present disclosure;

[0049] Figure 19 is a schematic diagram of the convolution operation after conversion on the basis of Figure 18 ;

[0050] Figure 20 is a schematic diagram of arrangement of multi-batch HWC88 storage format under implicit storage in one embodiment of the present disclosure;

[0051] Figure 21 This is a schematic diagram of the arrangement of HWC88 storage format of multiple batches in explicit storage in one embodiment of this disclosure;

[0052] Figure 22 This is a schematic diagram of the arrangement of HWC88 storage format in a semi-explicit storage in one embodiment of this disclosure;

[0053] Figure 23 This is a schematic diagram of the workflow of the HWC88 format automatic padding method in one embodiment of this disclosure;

[0054] Figure 24 This is a schematic diagram of an embodiment of the present disclosure, analogous to the case of automatic padding in multiple batches (1). Detailed Implementation

[0055] To further describe the present invention, the following description is in conjunction with the accompanying drawings. Figures 1A to 24 Further explanation is needed.

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below in conjunction with the embodiments of this disclosure. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0057] Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely to illustrate selected embodiments of the disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0058] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0059] In the description of the present disclosure, it needs to be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present disclosure.

[0060] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present disclosure, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified and limited.

[0061] In the present disclosure, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting", "fixing" and the like should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral; it can be directly connected, or indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present disclosure can be understood according to the specific circumstances.

[0062] In the present disclosure, unless otherwise explicitly specified and limited, the first feature "on" or "under" the second feature can include that the first and second features are in direct contact, or that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, the first feature "on", "above" and "on" the second feature includes that the first feature is directly above and obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature "under", "below" and "under" the second feature includes that the first feature is directly below and obliquely below the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0063] The present disclosure is described in detail below.

[0064] 1 Padding and related concepts

[0065] In convolutional neural networks, convolution refers to the process of calculating the input feature map (ifm) and the weight (wt, also known as the kernel) to obtain the output feature map (ofm). When there is no need to specifically distinguish between the input and output feature maps, ifm and ofm can be collectively referred to as the feature map (fm). Convolution operations include standard convolution, depthwise separable convolution, group convolution, and dilated convolution.

[0066] To ensure that ofm has the same width w and height h as ifm after the convolution operation, padding is involved.

[0067] like Figure 1A The white bars around the ifm shown are the padding added by ifm during standard convolution operations. In this example, the dimensions of ifm are 2 (height) * 24 (width) * 16 (channels), and the dimensions of wt are 3 (height) * 3 (width) * 16 (channels) * 16 (n), where n is the number of kernel groups. As you can see, ifm has a row of padding data at the top and bottom, and a column of padding data on the left and right. After ifm is padded with padding, the resulting ofm after convolution has the same width and height as ifm.

[0068] For example Figure 1B As shown, the dimensions of ifm are 2 (height) * 24 (width) * 16 (channel), and the dimensions of wt are 5 (height) * 5 (width) * 16 (channel) * 24 (n). It can be seen that ifm is padded with two rows of padding data at the top and bottom, and two columns of padding data on the left and right. After the ifm is padded, the convolution operation results in ofm with the same width and height as ifm.

[0069] 2 HWC88 storage format

[0070] The naming of the standard HWC storage format is based on the order of data arrangement in each dimension (Height, Width, Channel), and is arranged according to the priority from right to left. HWC88 indicates that the number of channels in the Channel direction is a multiple of 8, and the Width direction is divided into 8 equal parts.

[0071] An exemplary HWC88 storage format is as follows:

[0072] A row of input data fm is segmented as shown in Figure 2 The data between the two gray blocks is a segment of data to be operated on. In the Width direction, each segment of data is evenly divided into 8 blocks, and in the Channel direction, the data is divided into 0~ci segments (ci = Channel / 8) based on 8. The data of the same position and channel group in each block is stored in the same address in the memory, and the pixel points of the same position in each block are stored first.

[0073] As shown in Figure 2 In this example, each block of data contains 6 numbers in the Width direction and 8 numbers in the Channel direction.

[0074] For more detailed definitions and explanations of the HWC88 storage format, see the following text of the specification.

[0075] 3 HWC88 format automatic padding mechanism

[0076] Figure 2 As shown in the storage of fm in HWC88 format in the memory, assuming that the wt dimension for convolution operation is 3(H)*3(W)*8(C)*8(N), 1 column of padding data is needed on the left and right sides of the fm storage data.

[0077] For more detailed definitions and explanations of the HWC88 storage format, see the following text of the specification. According to the HWC88 storage format definition and Figure 13 There can be four cases when operating on the segmented ifm:

[0078] (1) [part_M_head] is the starting address of the row of fm, and [part_M_tail] is the ending address of the row of fm, i.e. the segment is exactly a complete row of fm or image;

[0079] (2) [part_M_head] is not the starting address of the row of fm, and [part_M_tail] is not the ending address of the row of fm, i.e. the segment is located in the middle of the image or fm;

[0080] (3) [part_M_head] is not the starting address of the row of fm, and [part_M_tail] is the ending address of the row of fm, i.e. the segment is the rightmost segment of the fm or image;

[0081] (4) [part_M_head] is the starting address of the fm in this line, and [part_M_tail] is not the ending address of the fm in this line, that is, this segment is the leftmost segment of fm or image;

[0082] When using IFM data stored in HWC88 format, during convolution operations, the 64 bytes of data in the same unit are sent to 8 multiplication units for convolution in each cycle. Each multiplication unit is responsible for 8 channels (8 bytes) of one point. Figure 3 A group of channels within a single cell. Each computational unit performs the entire convolution operation by accumulating the results of multiple cycles of computation on all points involved in the convolution.

[0083] For the ifm data in the part_M memory segment:

[0084] (1) The points involved in the convolution calculation in the middle part of memory, such as w=7, 8, 9, have their data originally stored in the same location in different storage units, and can be directly used for multiplication and accumulation;

[0085] (2) Points involved in convolution calculations on both sides of memory, such as w=5, 6, 7 or w=10, 11, 12, although the data are all in the part_M memory segment, some data are stored in different locations in different storage units. By reading and shifting the original data, they can be adjusted to the same position for multiplication and accumulation;

[0086] (3) The convolution points at the two ends of part_M in ifm, such as w=47(part_M-1), w=0(part_M), w=1(part_M) or w=46(part_M), w=47(part_M), w=0(part_M+1), cannot be obtained by simple reading and shifting because some data is not in the memory segment part_M. The processing method is related to the segmentation of ifm.

[0087] like Figure 3 As shown in the diagram, the padding data on both sides of the memory originates from the dashed arrows. After reading and shifting the original data, the empty bits are padded to fill the data columns before and after the fm segment, respectively. Figure 13 The data shown is at the addresses [part_M-1_tail] (part_M-1, w=47) and [part_M+1_head] (part_M+1, w=0).

[0088] This disclosure applies different padding methods to the above four situations, and the corresponding padding data is stored explicitly, implicitly, or semi-explicitly, respectively.

[0089] Explicit storage means that the storage units of the padding data on the left and right sides are explicitly placed in the memory, which needs to occupy additional memory units. Implicit storage means that the storage units of the padding data on the left and right sides are no longer placed separately in the memory, and the corresponding padding data is obtained by reading and shifting the data at a specific address in the memory during calculation. Semi-explicit storage means that the padding data on one side uses explicit storage and the padding data on the other side uses implicit storage.

[0090] Take the fm as an example: Figure 13

[0091] For case (1), that is, a piece of fm is exactly a complete row of fm or image, there is no part_M-1 and part_M+1, and the left and right are real padding values P, which are stored as implicit storage, and the actual arrangement mode is as shown in Figure 4 , that is, reading the data from the source position and shifting; wherein, the data at the [part_M-1_tail] and [part_M+1_head] addresses filled by the spare bits is the real padding value P.

[0092] Figure 4 The padding data in the dashed box is not stored in the memory, and the corresponding fm needs to be taken out as virtual padding storage by address calculation; compared with the source data storage, the virtual padding storage will produce up and down shift.

[0093] The data reading implementation scheme of the padding is as follows:

[0094] As shown in Figure 4 , assuming that the fm address to be read calculated by the control logic is fm_addr, and the current fm_addr position is identified by padding_left and padding_right.

[0095] When fm_addr is between padding_left and padding_right, it means that the ifm calculated in the current loop needs to read the data in the middle area as shown in Figure 4 , and the read data does not need to be shifted up or down.

[0096] When fm_addr is less than padding_left, it means that the ifm calculated in the current loop needs to read the data in the left padding area as shown in Figure 4 , and the read data needs to be shifted up.

[0097] ​When fm_addr is greater than padding_right, it means that the ifm calculated in the current loop needs to read the right padding region data as shown in Figure 4

[0098] The address offset between two points adjacent in the Width direction on the fm is C / 8*64, denoted as block_offset; the total address offset of a segment of fm in the Width direction is W / 8*block_offset, denoted as padding_offset; in this example, W=48 and C=8, so block_offset=64 and padding_offset=384.

[0099] The actual reading address is offset to the address of the data source in the memory by padding_offset, and the corresponding data is read out. The address calculation and padding data shift flag pseudo code for automatic padding are as follows:

[0100] if (fm_addr < padding_left)

[0101] fm_addr += padding_offset

[0102] padding_type = 1 / / 1 – left

[0103] else if (fm_addr > padding_right)

[0104] fm_addr −= padding_offset

[0105] padding_type = 2 / / 1 – right

[0106] else

[0107] fm_addr = fm_addr

[0108] padding_type = 0 / / 0 – middle

[0109] For case (2), the segment of fm is the middle part of the segmentation, and part_M-1 and part_M+1 exist on the left and right sides. The padding needed on the left side of part_M is the rightmost point [part_M-1_tail] of part_M-1, and the padding needed on the right side of part_M is the leftmost point [part_M+1_head] of part_M+1. From Figure 5B ​As can be seen, the gray areas represent the padding data needed on the left and right sides. However, this data originates from two separate memory locations and cannot be obtained within a single cycle through reading and shifting. Therefore, in this case, the padding storage is explicit, and the actual arrangement is as follows: Figure 5A As shown, similar to implicit padding storage, padding storage will have a vertical offset relative to its source data storage; simultaneously, the values ​​of the previous segment [part_M-1_tail] and the next segment [part_M+1_head] are copied as part of the current segment, respectively, to the beginning and end addresses, thus organizing the data into... Figure 5B The gray portion is formatted and stored at the beginning and end addresses of the current segment. During convolution operations, the padding portion is treated as normal data and no additional padding address needs to be calculated.

[0110] For cases (3) and (4), the segment fm is the leftmost or rightmost part of the segment, containing either part_M-1 or part_M+1. In these two cases, the padding storage is semi-explicit, and the actual arrangement is as follows: Figure 6 As shown, similar to implicit storage, copying the data from the source location will result in vertical offsets in the data stored in the source location.

[0111] For the side with fm data, the value at the address [part_M-1_tail] or [part_M+1_head] is copied as part of the current segment to the beginning or end address; the padding part on this side is treated as normal data and no additional padding address needs to be calculated.

[0112] On the other side, padding is added using an implicit padding method. This data is not stored in memory; it needs to be retrieved via address calculation and used as virtual padding storage. The implementation scheme for reading padding data on this side is as follows:

[0113] Case (3): When fm_addr is greater than padding_right, it means that the ifm calculated in the current loop needs to read as follows: Figure 6 In the case shown (3), the data in the padding area on the right side needs to be shifted down.

[0114] The actual read address is offset to the memory address where the data source is located by directly using `padding_offset`. The corresponding data is read out, and the pseudocode for automatically adding padding and calculating the address and padding data shift flag is as follows:

[0115] if (fm_addr > padding_right)

[0116] fm_addr -= padding_offset

[0117] padding_type = 2 / / 1 - right

[0118] else

[0119] fm_addr = fm_addr

[0120] padding_type = 0 / / 0 - middle

[0121] Case (4): When fm_addr is less than padding_left, it means that the ifm calculated in the current loop needs to read the data in the left padding area as shown in case (4), and the read data needs to be moved up. Figure 6

[0122] The padding_offset directly shifts the actual read address to the address of the data source in the memory, and the corresponding data is read out. The address calculation and padding data shift flag of the automatic padding are as follows:

[0123] if (fm_addr < padding_left)

[0124] fm_addr += padding_offset

[0125] padding_type = 1 / / 1 - left

[0126] else

[0127] fm_addr = fm_addr

[0128] padding_type = 0 / / 0 - middle

[0129] In combination with the definitions one to three of the context HWC88 storage format, the HWC88 format fm left and right padding about multi-batch:

[0130] According to the multi-batch HWC88 storage format definition and its schematic diagram, i.e. Figure 17 , there are also four cases when operating on the ifm segment, corresponding to three storage methods:

[0131] (1) The segment is exactly a complete row of fm or image;

[0132] ​(2) The segment is in the middle of image or fm;

[0133] (3) The segment is the rightmost segment of fm or image;

[0134] (4) The segment is the leftmost segment of fm or image.

[0135] Take fm as an example: Figure 17

[0136] For case (1), that is, a segment of fm is exactly a complete line of fm or image, there is no part_M-1 and part_M+1, and the left and right are real padding values, denoted as P, and the padding is stored as implicit storage. The actual arrangement is as shown in Figure 20

[0137] The data reading implementation scheme of padding is as follows:

[0138] Suppose the fm address to be read calculated by the control logic is fm_addr, and the current fm_addr position is identified by padding_left and padding_right.

[0139] When fm_addr is between padding_left and padding_right, it is indicated that the ifm calculated in the current loop needs to read the middle area data as shown in Figure 20

[0140] When fm_addr is less than padding_left, it is indicated that the ifm calculated in the current loop needs to read the left padding area data as shown in Figure 20

[0141] When fm_addr is greater than padding_right, it is indicated that the ifm calculated in the current loop needs to read the right padding area data as shown in Figure 20

[0142] The address calculation and padding data shift flag pseudo code of automatic padding are as follows:

[0143] if (fm_addr < padding_left)

[0144] ​​​​​fm_addr += padding_offset

[0145] padding_type = 3 / / 3 - all padding data

[0146] else if (fm_addr > padding_right)

[0147] fm_addr -= padding_offset

[0148] padding_type = 3 / / 3 - all padding data

[0149] else

[0150] fm_addr = fm_addr

[0151] padding_type = 0 / / 0 - middle

[0152] For case (2), this segment fm is the middle part of the segment, and part_M-1 and part_M+1 exist on both sides. The padding needed on the left side of part_M is the rightmost point [part_M-1_tail] of part_M-1, and the padding needed on the right side of part_M is the leftmost point [part_M+1_head] of part_M+1. In this case, the padding is explicitly stored, and the actual arrangement is as shown in Figure 21 , where the left padding is the value of the address of the last segment [part_M-1_tail] of each of the 8 fms, and the right padding is the value of the address of the next segment [part_M+1_head] of each of the 8 fms. When performing convolution operations, the padding part is treated as normal data, and no additional padding address calculation is needed.

[0153] For cases (3) and (4), this segment fm is the leftmost or rightmost part of the segment, and part_M-1 or part_M+1 exists. In these two cases, the padding is stored as semi-explicit storage, and the actual arrangement is as shown in Figure 22 .

[0154] For the side with fm data, the padding needed to be filled in the rightmost side of part_M is the leftmost point of part_M+1 [part_M+1_head], and the padding needed to be filled in the leftmost side of part_M is the rightmost point of part_M-1 [part_M-1_tail]. The padding part of this side is regarded as normal data, and no additional padding address calculation is needed;

[0155] The other side is filled with padding according to the implicit padding data method. The implementation scheme of reading the data of the padding of this side is as follows:

[0156] Case (3): When fm_addr is greater than padding_right, it means that the ifm calculated in the current loop needs to read the data in the right padding area of case (3) as shown in Figure 22 All data are filled with padding values. Whether the address offset matters or not does not matter, but to keep consistent with the previous method, the padding_offset is also offset. The address calculation and padding data shift flag pseudo code of automatic padding are as follows:

[0157] if (fm_addr > padding_right)

[0158] fm_addr -= padding_offset

[0159] padding_type = 3 / / 3 – all padding data

[0160] else

[0161] fm_addr = fm_addr

[0162] padding_type = 0 / / 0 – middle

[0163] Case (4): When fm_addr is less than padding_left, it means that the ifm calculated in the current loop needs to read the data in the left padding area of case (4) as shown in Figure 22 All data are filled with padding values. Whether the address offset matters or not does not matter. The address calculation and padding data shift flag pseudo code of automatic padding are as follows:

[0164] if (fm_addr < padding_left)

[0165] fm_addr += padding_offset

[0166] padding_type = 3 / / 3 - all padding data

[0167] else

[0168] fm_addr = fm_addr

[0169] padding_type = 0 / / 0 - middle

[0170] Figure 23 is the workflow of the automatic padding method of HWC88 format. Except for the gray box, which is configured by software parameters, the rest can be automatically completed by hardware.

[0171] 4 Advantages of automatic padding of HWC88 format

[0172] The main advantages of the automatic padding method based on HWC88 format are:

[0173] (1) In the three cases of implicit or semi-implicit storage of padding data, a part of the fm storage space can be saved.

[0174] In the case of implicit storage of padding, the storage space saved is usually:

[0175]

[0176] In the case of semi-implicit storage of padding, the storage space saved is:

[0177]

[0178] where wt_w is the width of wt, and wi is the width of each block after a row of fm is evenly divided into 8 blocks as mentioned above. Take the fm as an example, wi = 6, and if wt_w = 3, in the case of implicit access padding, 25% of the fm storage space is saved, and in the case of semi-implicit storage of padding, 12.5% of the storage space is saved. Figure 2

[0179] In special cases, the storage space saved may be reduced, such as Example 4.

[0180] (2) The overhead of moving part of the padding data is reduced, and the convolution operation efficiency is improved. In the case of implicit storage of padding data, the data movement on both sides can be saved; in the case of semi-implicit storage of padding data, the data movement on one side can be saved.

[0181] ​The following is described by way of more detailed examples:

[0182] 1. Different sizes wt

[0183] When the wt of the convolution operation is larger, in order to keep the width and height of the OFM after the convolution operation the same as that of the IFM, it is necessary to pad the image or IFM data with multiple rows and columns of padding. The automatic padding method of the present invention is also applicable to this situation. It still copies and shifts the original data on the left and right sides of the data stored in the FM, and then pads the empty spaces with padding data before and after the data columns of the FM segment.

[0184] like Figure 7B As shown, taking an fm dimension of 2 (height) * 48 (width) * 8 (channel) and a wt dimension of 5 (height) * 5 (width) * 8 (channel) * 8 (n) as an example, ifm is padded with two rows of padding, analogous to the aforementioned case (1), the padding data is implicitly stored, and two columns of padding data are added to the left and right sides of each row of fm stored data; the data source is as follows Figure 7B As shown by the dashed arrow, after copying and shifting the original data, any empty bits are filled with the actual padding value P. At this point, block_offset = C / 8*64 = 64; padding_offset = W / 8*block_offset = 384. We can set padding_left = 0 and padding_right = 383. If the current loop calculates addr = -64, then the actual data address is addr + padding_offset = -64 + 384 = 320, and the retrieved data needs to be shifted up; if the current loop calculates addr = 448, then the actual data address is addr - padding_offset = 448 - 384 = 64, and the retrieved data needs to be shifted down; if the current loop calculates addr = 192, then no address correction is needed, and the retrieved data does not need to be shifted up or down.

[0185] If the above example is a segment of a row of data, and the data source is as follows: Figure 7AAs shown by the dotted arrows, the original data is copied and shifted, and the data in the empty bits is filled in the data columns before and after the fm segmentation, [part_M-1_tail], [part_M-1_tail-1] and [part_M+1_head], [part_M+1_head+1] positions. Wherein, [part_M-1_tail-1] represents the second last point of part M-1, and [part_M+1_head+1] represents the second point of part M+1. The above-mentioned cases (2), (3) and (4) can be analyzed by analogy, and explicit storage or semi-explicit storage can be used accordingly.

[0186] 2 Different Channel Numbers

[0187] Suppose that the dimension of fm is 2 (height) * 48 (width) * 64 (channel), and in the channel direction, it is divided into ci segments (ci = C / 8) with 8 as the base; when storing fm in HWC88 format, the data of the same position and the same channel group of each block is placed in the same address of the memory, and all channels of the same position pixel point in each block are stored preferentially.

[0188] Padding is stored as implicit storage, and the actual arrangement mode is as shown in Figure 7C , wherein the padding part is to copy and shift the data from the source position. In Pa.b, a represents the bit order in the width direction, and b represents the bit order in the channel direction.

[0189] At this time, block_offset = C / 8*64 = 64 / 8*64 = 512; padding_offset = W / 8*block_offset = 48 / 8*512 = 3072. It can be set that padding_left = 0 and padding_right = 3071. If the addr calculated in the current loop is -64, the actual data address is addr+padding_offset = -64+3072 = 3008, and the data taken needs to be moved up; if the addr calculated in the current loop is 3100, the actual data address is addr-padding_offset = 3200-3072 = 128, and the data taken needs to be moved down; if the addr calculated in the current loop is 3008, the address does not need to be corrected, and the data taken does not need to be moved up or down.

[0190] If the above example is a segment of a row of data, and C is any multiple of 8, as shown in Figure 8As shown, the padding data on the left and right sides are the data columns before and after the fm segment, which are filled with empty spaces after copying and shifting the original data. P-1.0, P-1.1...P-1.c-1 and P48.0, P48.1...P48.c-1 refer to the last point of the previous part_M-1 segment or the first point of the next part_M+1 segment. It can also be analyzed by analogy with the above cases (2), (3) and (4), and explicit storage or semi-explicit storage can be used accordingly.

[0191] The width of 3fm is 8

[0192] Assuming the dimensions of fm are 1 (height) * 8 (width) * 8 (channel) and the dimensions of wt are 5 (height) * 5 (width) * 8 (channel) * 8 (n), padding needs to be added to the top, bottom, left, and right sides of ifm, with 2 rows or 2 columns of padding.

[0193] like Figure 9 As shown, in this embodiment, the real data is stored in HWC88 format and only occupies one memory unit. It is impossible to directly complete the automatic padding using the aforementioned situation (1) and the method of embodiment 1. In this invention, the first step is to explicitly pad a set of padding data on the left and right sides of fm in memory according to the explicit storage method, so as to obtain the explicitly stored fm that occupies 3 storage units.

[0194] Then as Figure 10 As shown, following the implicit storage method, the above-mentioned fm is then supplemented with the second group of padding, and its data source is as follows: Figure 10 As shown by the dashed arrow, after copying and shifting the original data, empty spaces are padded with actual padding. This allows the padding value to be obtained automatically during convolution operations. In this case, because some data needs to be explicitly stored, the storage space saved is only 40%.

[0195] 4. Multiple batches

[0196] like Figure 24As shown, taking the example of the fm dimension being 1 (height) * 48 (width) * 8 (channel) and the wt dimension being 3 (height) * 3 (width) * 8 (channel) * 8 (n), the case of multiple batch automatic padding (1) is analogized. Padding data is stored implicitly, and one column of padding data is added to the left and right of each row of fm stored data. The left and right sides are supplemented with real padding values P. To maintain consistency with the previous method, block_offset = C / 8*64 = 64 is also calculated. padding_offset = W*block_offset = 3072. It can be set that padding_left = 0 and padding_right = 3071. If the addr calculated in the current loop is -64, which is less than padding_left, the actual data address is addr + padding_offset = -64 + 3072 = 3008, and the data is filled with padding values. If the addr calculated in the current loop is 3072, the actual data address is addr - padding_offset = 3072 - 3072 = 0, and the data is filled with padding values.

[0197] If the above example is a segment of a row of data, cases (2), (3), and (4) of multiple batch automatic padding can also be analogized and explicit storage or semi-explicit storage can be used accordingly.

[0198] It is noted that, with respect to the HWC 88 storage format disclosed in the present disclosure, It is used to realize the rearrangement of the FM organization format in data storage, and needs to meet the following conditions: the format has good locality, can improve the utilization rate of data in the cache, and can improve the data transmission bandwidth and computing performance.

[0199] For example, the HWC88 storage format is specifically:

[0200] The FM data in a row or a part of a row is evenly divided into 8 blocks, and the data in the same position and the same channel group of each block is stored in the same address of the memory, and all channels of the same position pixel points in each block are stored preferentially.

[0201] For example, the FM adopts the HWC88 format for convolution operation, which is specifically: the last row of FM data is evenly divided into 8 blocks to obtain 8 parallel points, and the data in the same position and the same channel group of each block is multiplied by the corresponding channel of the convolution kernel.

[0202] For example, FM uses the HWC88 format for depthwise convolution operations as follows: the 8 parallel points on FM are multiplied with the corresponding channels of the convolution kernel, the convolution kernel is copied into 8 copies and stored in the Block Mem storage area for computation, and the data format before and after copying is the HWC8 format.

[0203] In other words, the HWC88 format disclosed in this disclosure is a new data storage format defined independently based on the HWC format.

[0204] Definition 1: Select 8 points with different W values ​​in the same row of the feature map. Each point has 8 channels, stored in one memory unit (64 bytes), and C is limited to a multiple of 8, and W is also a multiple of 8. This explains the meaning of the last two "8"s in HWC88 compared to the HWC format, and provides one definition of the HWC format.

[0205] From another perspective, for FM data, the HWC88 format divides a line or a portion of a line of FM data into 8 equal blocks. Each block contains wi (wi = W / 8) data points. Then, the data of the same channel group (8 channels per group) at the same position in each block, a total of ci groups (ci = C / 8), are placed at the same memory address. Priority is given to storing all channels of the same pixel in each block.

[0206] like Figure 11 The image shows FM data stored in HWC88 format.

[0207] An example of HWC88: Figure 12 As shown, in this example, W=48 and C=16. After splitting in the W and C directions, each block of data contains 6 numbers in the W direction (wi=6) and 2 groups in the C direction (ci=2).

[0208] The advantage of the HWC88 data storage format is that it has fewer restrictions on the C value compared to the HWC format, saving data storage space in most cases where padding with zeros is required. For example, for an RGB image with C=3, if W=60 and H=1, storing it in the HWC format requires 60 memory units; while storing it in the HWC88 format only requires 8 memory units.

[0209] Furthermore, the HWC88 format disclosed in this disclosure can also be defined as follows:

[0210] Definition 2: For FM data, firstly, the entire input data fm is divided into segments, such as... Figure 13As shown, the data between the two baffles in the figure represents a segment of data to be processed. Then, following the method described in Definition 1, in the W direction, this segment of data fm is divided into 8 equal blocks, each containing wi (wi = W / 8). In the C direction, it is divided into ci groups based on 8 (ci = C / 8). Data from the same position and channel group in each block is placed in the same memory unit, prioritizing the storage of all channels of pixels at the same position in each block. In the example figure, Wp = 48, C = 8, and the values ​​are variable. According to the requirements of the HWC88 format, both Wp and C are multiples of 8.

[0211] To distinguish them, the HWC88 format described in Definition 1 is called the basic HWC88 format, and the HWC88 format described in Definition 2 is called the improved HWC88 format.

[0212] like Figure 14 As shown, in this example, W=144, C=16, after being divided into 3 segments, each segment has Wp=48. After being divided in the W and C directions, each block of data contains 6 numbers in the W direction (wi=6) and 2 groups in the C direction (ci=2).

[0213] The advantage of the improved HWC88 format is that once the algorithm network is determined, the limitations on the image or feature map size are smaller. Taking a feature map with W=712 as an example, if stored in the basic HWC88 format, the wi of each block is calculated (wi=W / 8=712 / 8=89), and then data from the same position and channel group in each block are placed in the same memory unit. However, this block wi may not be suitable for the convolution operations of subsequent layers of the network, which will introduce additional data storage format conversion requirements. With the improved HWC88 format, the Wp of each segment and the corresponding wi of the blocks within the segment can be determined from the algorithm before the blocks are divided. For example, if Wp=128 (corresponding to wi=128 / 8=16), then the segments are divided according to Wp=128 for each segment. The first 5 segments are complete segments, and the last segment has only 72 valid points. The missing parts are padded with 0, and then the data within the segment is divided into blocks, and data from the same position and channel group in each block are placed in the same memory unit.

[0214] The key point is that the HWC88 format, regardless of its definition, is used to implement the rearrangement of the format of FM in data storage, and needs to meet the following conditions: the format has good locality, can improve the utilization of data in the cache, and improve data transmission bandwidth and computing performance.

[0215] Further, it can be found that the basic HWC88 format is a special case of the improved HWC88 format, and one row of data of fm is divided into one segment, and Wp=W. The improved HWC88 format is an extension of the basic HWC88 format, especially for a larger size FM.

[0216] In another embodiment, 8 channels are a group.

[0217] In another embodiment, the FM adopts the HWC88 format for convolution operation, specifically: the last row of FM data of the FM is evenly divided into 8 blocks to obtain 8 parallel points, and the data of the same position and the same channel group of each block is multiplied by the corresponding channel of the kernel.

[0218] In this embodiment, the weight data (i.e. WT data) is stored in the memory according to the HWC88 format, but the format of the weight will be different according to the FM data format and the type of calculation during calculation. Generally, the WT data is offline and has a good format.

[0219] If the FM adopts the HWC88 format for convolution operation, the parallel degree in the CH direction is 8, that is, 8 channels are a group, and the parallel degree in the N direction is also 8, that is, 8 kernels are a group. When storing the WT, 8 channels of a group of kernels corresponding to a point are placed in one address, as shown in the following table: Figure 15 The first group of 8 kernels P0 points (H=0, W=0) of 4 groups of channels are stored first, and then the data of the second group of 8 kernels is stored.

[0220] If all channels of points with the same w value in Figure 14 are combined into one cell, and only the w value is marked in the cell, then Figure 14 the storage format of the data in the memory can be simplified to the representation of Figure 16 .

[0221] Further, the HWC88 storage format can also have three definitions:

[0222] If there are multiple images for parallel calculation during convolution operation, first, 8 images to be calculated in parallel are taken, and each row of input data fm of each image is segmented, as shown in the following table: Figure 17 The data between the two gray baffles is one segment of data to be operated, and then each image is divided into ci groups (ci=C / 8) in the C direction with 8 as the base, and the data of the same position and the same channel group of each image is placed in the same unit of the memory, and all channels of the same position in each block are stored first. Figure 17In the part_M-1 and part_M+1 sections, all channels with the same n and w values ​​(such as all channels with n=7 and w=47 as shown in the dashed box) have been merged into a single simplified representation. This HWC88 format is called the multi-batch HWC88 format, and its advantage is that there are no restrictions on the W direction of fm, as long as C is a multiple of 8.

[0223] Based on the three definitions of the HWC88 storage format described above, this disclosure further describes in detail the automatic padding mechanism of the HWC88 format:

[0224] According to the definition of convolution operation, such as Figure 18 As shown, when calculating H=0, W=0 (denoted as P(0, 0)) in ofm, the 9 points in the left frame of ifm, the 5 padding points above and to the left, and P(0, 0), P(0, 1), P(1, 0), and P(1, 0) are used. Similarly, when calculating P(2, 2) in ofm, the 9 points in the middle frame of ifm, P(1, 1), P(1, 2), P(1, 3), P(2, 1), P(2, 2), P(2, 3), and the 3 padding points below are used; when calculating P(1, 5) in ofm, the 9 points in the right frame of ifm, P(0, 4), P(0, 5), P(0, 6), P(1, 4), P(1, 5), P(1, 6), P(2, 4), P(2, 5), and P(2, 6) are used. Figure 18 In the diagram, each cell in ifm represents a point and contains all its channels; each cell in ofm represents a point and contains one or more channels, determined by the specific convolution operation, and does not affect the discussion of the automatic padding mechanism.

[0225] The padding operations involved in convolutional neural network operations include padding the feature map vertically and horizontally.

[0226] Top and bottom padding:

[0227] Still with Figure 18 For example, when calculating P(2, 2) and other points in the same output row in ofm, padding is involved. Since the padding values ​​are all 0, the sum of their multiplication with wt is also 0. Therefore, the software can control this calculation to not be performed, i.e., convert it to... Figure 19 The convolution operation is shown.

[0228] Basic / Improved HWC88 format FM left and right padding:

[0229] Because the interline data is originally independent of each other due to the storage in the HWC88 format, the discussion is made by taking one line of data as an example, and the rest lines can be analogized.

[0230] In summary, the disclosure proposes a hardware automatic padding strategy and implementation of convolution operation. Padding data is automatically supplemented on the left and right sides of data in the HWC88 storage format by copying and shifting the original data.

[0231] Although the embodiments of the disclosure are described above with reference to the drawings, the disclosure is not limited to the specific embodiments and application fields described above, and the specific embodiments described above are only illustrative and guiding, but not limiting. A person of ordinary skill in the art can make many forms under the inspiration of the present disclosure and without departing from the scope protected by the claims of the disclosure, which all belong to the protection of the disclosure.

Claims

1. A method for automatic hardware padding in convolution operations, characterized in that: The method uses an HWC88 data storage format to store the data, wherein, In this data storage format, the entire input data ifm is segmented during the data storage process. Each segment is divided into 8 equal parts to obtain 8 sets of data. Then, each set of data is extracted at equal positions, and the data at equal positions are arranged in a column and stored sequentially. The method involves automatically supplementing padding data on both sides of segmented data using the HWC88 data storage format by copying and shifting the original data of the data columns before and after the current segment, and then performing convolution operations. In the case of explicitly storing padding, the padding part is treated as normal data and no additional padding address needs to be calculated. The corresponding padding data is stored either explicitly or implicitly. Explicit storage means that the storage units containing the padding data on the left and right sides of the segment are explicitly placed in memory, requiring additional memory units. Implicit storage means that the storage units containing the padding data on the left and right sides of the segment are no longer placed in memory separately. During calculation, the corresponding padding data is obtained by reading and shifting data at specific addresses in memory.

2. The method according to claim 1, wherein, For different weight wt sizes of convolution operations, the method copies and shifts the original data of the data columns before and after the segmentation on both sides of the segmented data stored in the feature map fm, and then fills the empty spaces with padding data. That is, the data on both sides of the current segment are filled with padding data by the data columns before and after the segmentation.

3. The method according to claim 1, wherein, For different channel numbers When the feature map fm has dimensions height*width*channel of 2*24*C, it is divided into segments from 0 to ci in the channel direction with a base of 8, where ci = C / 8 and C is the number of channels; when storing fm in HWC88 format, the data of the same channel group at the same position in each block are placed at the same address in memory, and all channels of the same pixel in each block are stored first. Assuming a column width of memory is 64 bytes, it can store data for exactly 8 points, with each point having 8 channels. Then, the address offset between two adjacent points in the width direction of fm is 64 bytes * ci, denoted as block_offset. When padding is stored implicitly, the padding portion read is the data from the data columns before and after the segmentation, which is copied and shifted.

4. The method according to claim 1, wherein, When the feature map fm has a dimension of height * width * channel of 1 * 8 * 8 and the feature map wt has a dimension of height * width * channel * n of 5 * 5 * 8 * 8, where n is the number of kernel groups, ifm needs to be padded with 2 columns of padding when stored in HWC88 format; For cases where the segment has only one column after being stored in HWC88 format, the segment data is first copied and shifted according to the explicit storage method. Then, a set of padding data is added to both sides of fm to obtain fm that occupies 3 columns of explicit storage. Then, following the implicit storage method, the data columns before and after the above-mentioned 3-column display storage fm segment are copied and shifted to supplement the second column of padding on the left and right sides of the above-mentioned 3-column display storage fm.

5. The method according to claim 1, wherein, The HWC88 data storage format is specifically as follows: The feature map (fm) data of a row or a portion of a row is divided into 8 blocks. Data of the same channel group at the same position in each block is placed at the same memory address, prioritizing the storage of all channels of pixels at the same position in each block.

6. The method according to claim 1, wherein, The feature map fm uses the HWC88 format for convolution operations. Specifically, the previous row of fm data is divided into 8 blocks to obtain 8 parallel points. The data of the same position and the same channel group in each block are multiplied by the corresponding channel of the convolution kernel.

Images