Method and apparatus for max pooling of convolutional neural networks

Abstract

The present application relates to the field of computer vision and artificial intelligence, and specifically to the processing of the pooling layer of the convolutional neural network, and proposes a method and device for large-core pooling. The method mainly includes the HBLK / WBLK block mode of mapping the pooling output to the input, including: performing the operation of the pooling core kh / kw length characteristic data on the H / W direction dimension, saving the temporary results of the direction in the internal cache SRAM, storing the temporary data in the maximum output mode, and when the internal cache size range is exceeded, the H / W direction block of cblk is performed; the W and H directions represent the width and height directions of the feature map; kw / kh are the sizes of the W / H direction pooling core respectively. The device includes a pooling top layer control device, an input / output control device, an HBLK unit control device, an HBLK unit control device, and a pooling operation device. The present application can realize the operation of the pooling core with unlimited size, the hardware device proposed can achieve the balance of performance, bandwidth, power consumption and area, and can effectively solve the technical problems such as small pooling core size and high redundancy of tensor characteristic data operation in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and artificial intelligence, in particular to the processing of the pooling layer of the convolutional neural network, and proposes a large kernel pooling method and device. BACKGROUND

[0002] With the rapid development of the field of artificial intelligence, convolutional neural networks (CNN) are more and more applied to the fields of image classification and image recognition.

[0003] The convolutional neural network usually contains multiple groups of convolutional layers, pooling layers and other neural network layers. The convolutional layer can extract local features of data, and the pooling layer is used to reduce the parameter amount and the operation of the neural network. The pooling layer usually contains two operations: maximum value pooling and average value pooling operations.

[0004] At present, the industry is accelerating the landing application of AI chips, and the AI chip software and hardware architectures proposed by different manufacturers are also various. An AI chip that can balance the factors of computing power, power consumption, versatility, area, etc. can better occupy the application market.

[0005] The existing AI chip hardware structure including convolution kernel or pooling kernel mainly includes two categories. The first category adopts a customized instruction set architecture, and the hardware operation unit calculates under the instruction format by designing the smallest hardware unit (such as a convolution operation kernel, a pooling operation kernel). The advantage of this architecture is strong versatility and simple stacking of computing power, but the disadvantage is that the software is responsible for splitting instructions, and there are many redundant operations and data reading in the hardware, which has no advantage in bandwidth and power consumption; the second category such as the architecture of Google TPU, such as the hardware structure of the impulse circuit customization, has the advantages of less data redundancy in the operation process, high energy efficiency ratio, etc., and the disadvantage is that the hardware design control complexity is high, and some general specifications are not high, such as the large kernel size of the convolution kernel and the pooling kernel including kernel, stride, size, etc.

[0006] In summary, the AI chip pooling layer hardware circuit implementation faces the problems of versatility and energy efficiency ratio (proportion of power consumption, bandwidth, computing power and area), therefore, a pooling hardware method and device with good versatility and high energy efficiency ratio are needed SUMMARY

[0007] The present application aims to overcome the shortcomings of the prior art, and provides a convolutional neural network large kernel pooling method and device to effectively solve the technical problems of small pooling kernel size and high tensor feature data operation redundancy in the prior art.

[0008] To achieve the above object, the technical scheme of the present application is:

[0009] In the first aspect, the present application provides a method for large kernel pooling of convolutional neural network, which comprises:

[0010] The HBLK / WBLK block mode of the pooling output mapping to the input comprises:

[0011] The operation of the H / W direction dimension on the kernel kh / kw length feature data is carried out, and the temporary operation result of the direction is saved in the internal cache SRAM, which is stored in the maximum output temporary data amount, and when the internal cache size range is exceeded, the H / W direction block cblk is carried out; W and H represent the width and height directions of the feature map; kw / kh are the sizes of the W / H direction pooling kernel.

[0012] Further, the HBLK / WBLK block mode of the pooling output mapping to the input further comprises:

[0013] The operation range of the pooling kernel operation is trimmable, including the trimming of W, H and C directions, and when the internal HBLK / WBLK control unit determines that the block is needed according to the internal cache and the current pooling parameter, the HBLK / WBLK input unit information of the pooling operation device is automatically operated by the hardware device according to the output parameter and the current block dynamic coordinate information.

[0014] Further, the method further comprises:

[0015] When the size of the pooling kernel is greater than the maximum range [Kw_max, Kh_max] supported by the hardware device, a command CMD multiple splitting mode is defined, the configuration information of each splitting kernel is configured, and after multiple operations, the final pooling operation result is combined by partial kernel result operation to support the pooling kernel with kw>Kw_max or kh>Kh_max; Kw_max / Kh_max are the maximum lengths of the pooling kernel supported by the current pooling hardware device in the W / H direction.

[0016] Further, the method further comprises:

[0017] The pooling tensor feature data is stored externally, comprising:

[0018] The tensor feature data is divided in the C direction dimension with a cblk_size size; the C direction dimension is the channel dimension.

[0019] Further, data of the current cblk(n) is stored in a manner of W·(H·csize), and in the cblk(n), data is continuously stored in a manner of h(m)·cblk_size for one W plane, and is continuously saved to the last W plane of the current cblk(n), and then tensor data of the next cblk(n+1) block is stored; cblk(n) represents a block size (n>=1) of the C direction divided in a manner of cblk_size.

[0020] Further, the cblk_size is a power of 2.

[0021] Further, the method further comprises:

[0022] Storage of the pooling operation temporary result in the internal cache comprises:

[0023] The pooling core size parsed by the current CMD command is kw·kh, wherein kw / kh are the sizes of the W / H direction pooling core respectively, and Kw_max / Kh_max are the maximum lengths of the W / H direction supported by the current pooling hardware device;

[0024] Kw_max+1 single-port SRAMs are set as the internal cache, wherein the data width of a single SRAM is defined as cblk_size*d_size, wherein d_size is the bit width of the pooling feature data, the depth of the SRAM is set as sram_h, and a single SRAM can store the intermediate result data of one W plane sram_h*cblk_size*d_size in the current cblk.

[0025] Further, the storage of the pooling operation temporary result in the internal cache further comprises that the SRAM structure for storing temporary data is dynamically adaptive, when kw is less than a set threshold, the sram in the H direction is stored in a combined manner to increase the depth of the temporary data, so as to reduce the reading of redundant tensor feature data, and a larger pooling cblk block is operated.

[0026] In a second aspect, the present application provides a device for convolutional neural network large-core pooling, and the device comprises:

[0027] The pooling internal top-level control device is used for parsing command information configured from the external top-level control module, and according to the command information, the internal is divided into a loop control flow of batch(n)→cblk(m)→hblk(k), wherein n, m, k>=1, and the hblk(k) block is the smallest core operation unit thereof;

[0028] The pooling feature data input and output device is used for controlling reading or outputting of tensor data from an offline DRAM device or an online module, and the data block information is provided by the pooling internal top layer control device, and the data block size is one hblk block unit.

[0029] The HBLK unit control device is used for reading input buffer raw data, and controlling the pooling device to perform H direction -> W direction operation process, and finally outputting the pooling operation result to the output device.

[0030] The pooling operation device comprises maximum value and average value H and W direction operation circuits respectively, wherein the operation circuits of the two kinds of pooling operations in each direction share the same set of register buffer units, when the buffer data is greater than kh / kw, the corresponding pooling operation is performed on the effective data, and the pooling result is output.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] (1) A method of pooling tensor data in external storage and intermediate result in internal buffer data format is provided, the C direction dimension is divided according to cblk_size, the feature data is stored in the manner of h·cblk_size, and the internal buffer sram is matched, so that the chip energy efficiency and throughput speed can be maximized in hardware implementation;

[0033] (2) An HBLK block method of "output to input mapping" is provided, and a dynamic adaptive internal buffer sram array is designed. According to the hardware device operation HBLK unit, the hardware design complexity is reduced, and the redundant data operation is also minimized;

[0034] (3) A method of splitting large kernel pooling is provided, which has universality. According to the method of the present application, a certain sram size is configured in a single pooling device, which can easily support kernel greater than 32x32, 64x64, etc. When the range is exceeded, the method can split the kernel configuration multiple CMD to achieve the same effect, which can easily support the current deep learning network architecture pooling 7x7, 9x9 window. The device large kernel design scheme technical index has leading nature. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is the overall flowchart of the large kernel pooling method provided by the embodiment of the present application;

[0036] Figure 2 is a structure schematic diagram of a pooling command (CMD) tensor data input and output provided by the embodiment of the present application;

[0037] Figure 3is a storage format schematic diagram of the pooled feature data stored externally (DRAM medium) provided by the embodiment of the application.

[0038] Figure 4 is a structure schematic diagram of the "output mapping to input method" provided by the embodiment of the application.

[0039] Figure 5 is a parameter schematic diagram of the input and output in the W / H direction provided by the embodiment of the application.

[0040] Figure 6 is a HBLK block mapping method schematic diagram provided by the embodiment of the application.

[0041] Figure 7 is a pooled device H direction processing flow schematic diagram provided by the embodiment of the application.

[0042] Figure 8 is a temporary data organization structure schematic diagram of the pooled device in the sram provided by the embodiment of the application.

[0043] Figure 9 is a structure schematic diagram of the pooled device provided by the embodiment of the application.

[0044] Figure 10 is a pooled kernel according to the CMD split calculation method flow schematic diagram provided by the embodiment of the application. DETAILED DESCRIPTION

[0045] Embodiment:

[0046] In order to illustrate the purpose, advantage and overall technical scheme of the method and device of the application, the technical scheme of the application will be further described below in combination with the drawings and embodiments.

[0047] Firstly, some words and abbreviations appearing in the present application are explained:

[0048] (1) W / H / C respectively represent the three dimensions of the pooled tensor feature data, wherein W and H represent the width and height directions of the feature map, and C represents the channel dimension, wherein "_in", "_out" and "_blk" represent the input, output and block data unit of a certain dimension;

[0049] (2) kernel, stride and pad respectively represent the current pooling kernel size, sliding step and boundary padding size of the pooled tensor feature data, wherein according to the two dimension directions of W / H, kw / kh, sw / sh and pw / ph are defined as the corresponding parameters in the width and height directions, pad direction also includes pw_lft and pw_rgt representing the left and right boundary padding size of pw, and ph_abv and ph_bot representing the upper and lower boundary padding size of ph.

[0050] (3)batch(k) represents the batch number of input and output tensor feature data operated by one command CMD in one configuration into the pooling hardware device (k>=1); cblk(n) represents the block size in the C direction according to cblk_size (n>=1), wherein cblk_size represents the number of channels of the current cblk block, such as cblk_size=c16, which represents that the current cblk block is divided into blocks according to 16 channels;

[0051] (3) The internal cache sram_h represents the depth of each single-port SRAM, wherein d_size represents the bit width of each feature point data.

[0052] Figure 1 is the overall flowchart of the large nuclear pooling method of the present application, which mainly consists of four parts:

[0053] The pooling HBLK unit flowchart 104 is the core operation component of the present application, which proposes the HBLK blocking method of "mapping from output to input". The internal flowchart first performs pooling operation on the h·cblk data of the current hblk in the H direction to obtain the intermediate result stored in the SRAM cache. After completing the H direction operation of the kw h·cblk planes, the W direction pooling operation is performed. The whole process is carried out in a pipeline. It should be noted that the "HBLK blocking" method described in the present embodiment does not limit the blocking method to be performed in the H direction dimension. It is also applicable to blocking in the W direction dimension (WBLK blocking). The difference lies in that the storage format of the current cblk(n) in the memory is W·(H·csize) or H·(W·csize) format. Similarly, the "WBLK blocking" or the first "according to the W direction" splitting method similar to the embodiment of the present application also belongs to the protection scope of the present application.

[0054] The pooling CBLK unit flowchart 103 is the storage method of the tensor feature data in the external memory proposed by the present application. The feature data in the C direction (channel) is segmented. This segmentation method can balance the kernel size, internal operation unit and storage SRAM of the pooling device, can perform pooling operation according to a certain block capacity, and avoid too much continuous data in the C direction read, which limits the pooling kernel size of the internal cache.

[0055] The pooling batch unit flowchart 102 is mainly used for pooling input tensor feature data which is often in multiple batches, but shares the same parameters of the pooling kernel. The pooling process of multiple batches is automatically controlled by hardware, which reduces the overhead of software and hardware interaction, and the number of batches is at least 1.

[0056] The pooling kernel splitting command unit 101 is a large core pooling splitting method proposed by the present application. When the size of the pooling kernel is greater than the maximum range [Kw_max, Kh_max] supported by the hardware device, multiple pooling operations are performed by splitting the CMD, and the final result is obtained by combination.

[0057] The entire pooling process is represented by the pseudo code as follows:

[0058] for (cmd = 0; cmd < CMD_NUM; cmd = cmd + 1)

[0059] for (batch = 0; batch < BATCH_NUM; batch = batch + 1)

[0060] for (cblk = 0; cblk < CBLK_NUM; cblk = cblk + 1)

[0061] for (hblk = 0; hblk < HBLK_NUM; hblk = hblk + 1)

[0062] {HBLK pooling operation;}

[0063] After several loop calculation processes, the final pooling layer operation result is obtained.

[0064] Figure 2 The structure diagram of the pooling command (CMD) tensor feature data input and output provided by the embodiment of the present application. It shows the size change of the pooling tensor feature data from the input 201 to the output 202, wherein there is at least one batch input, each batch is divided into at least one cblk(n) unit according to cblk_size, the size of each cblk(n) is win / hin, after the operation of the pooling kernel, stride, pad and other parameters, the final result corresponding to wout, hout is obtained. Wherein out = ceil((in+pad0+pad1-k) / s)-1, wherein ceil() is the integer function, in / out is applicable to W, H direction, pad0 / pad1 is equal to pw_lft / pw_rgt or ph_abv / ph_bot, k is equal to kw, kh, and s is equal to sw, sh.

[0065] The pooling tensor data in the storage format of the DRAM is stored in a manner of W·(H·csize) in the current cblk(n), and the data is stored in a manner of h(m)·cblk_size in the cblk(n), and the W plane is continuously stored, and the last W plane of the current cblk(n) is continuously stored, and then the tensor data of the next cblk(n+1) block is stored. Figure 3 is an embodiment schematic diagram of the storage method. Figure 301 stores the feature data of the continuous current cblk from C0 (stored in the LSB) to Csize-1 (stored in the MSB) according to the current h(m) = H0, and when the Csize feature data is stored, the next batch of Csize pixels is stored according to the next h(m+1) = H1; when all the h(m)·cblk_size data of the W plane is stored, the data of the next W is continuously stored, and the data of the last W plane of the current cblk(0) is reached, as shown in Figure 302, and then the data of the next cblk(1) is stored, and finally the last cblk(n) of the current batch is reached, as shown in Figure 302.

[0066] Figure 4 is a structural schematic diagram of the "output mapping to input method" provided by the application. The pooling layer operation of the application supports the pooling process of the entire tensor feature data, and also supports a cropable region operation, and the cropable region operation is suitable for the ROI (region of interest) pooling process of deep learning, and is defined as wout_crop_len and hout_crop_len, which represent the lengths of the output wout and hout cropping, respectively. Through the mapping method of the HBLK block, the top control module and the input control device of the application read the block data of the input region. Figure 8

[0067] Figure 5 is a parameter schematic diagram of the input and output in the W / H direction provided by the application, and necessary parameters in the W and H directions are defined. As shown in Figure 501, the output region is defined as the cropped actual coordinates hout_st and wout_st, the cropped region size wout_crop_len*hout_crop_len, and the output size Hout and Wout of the original input tensor data, which are calculated according to the formula out = ceil((in+pad0+pad1-k) / s)-1; the input region contains a padding region, as shown in Figure 502, and the padding sizes above, below, left and right are ph_abv, ph_bot, pw_lft and pw_rgt, respectively; and Figure 503 is the hin_st and win_st coordinates calculated by the output mapping method provided by the application.

[0068] ​The "HBLK partitioning" method embodiment of the present application, in combination with Figure 6 is described as follows:

[0069] Step 1-1: As shown in FIG. 601, determine the hblk_size of the current output HBLK, which represents the partition height of the current HBLK partition. The value of hblk_size is determined by kw and the internal cache SRAM size provided by the current pooling device, which is simply represented as:

[0070] hblk_size = LUT(sram_buf_size, kw), where "LUT()" represents the current size determined by table lookup. hblk_size is dynamically adaptive, and when kw is smaller, sram size (represented by sram_buf_size) is larger, and hblk_size is larger. The detailed adaptive process will be described in detail below in combination with Figure 9 .

[0071] Step 1-2: Calculate the height hin_size_all mapped to the input HBLK, represented as:

[0072] hin_size_all = (hblk_size - 1) * sh + kh;

[0073] Step 2: Define the H-direction coordinate of the output HBLK as hout(n), according to the cropping pooling method, the initial value hout(0) = hout_st, which is first mapped to the input coordinate as: hin(n) = hout(n) * sh. Since the mapping region may be in the ph_abv padding region, both hin(n) and ph_abv need to be adjusted according to the current partition. Define the upper boundary coordinate of the partition cur_ph_abv(n), as shown in FIG. 602, and the calculation method is:

[0074] cur_ph_abv(n) = (hout(n) * sh < ph_abv)? 0 : (hout(n) * sh - hin(n));

[0075] The H coordinate mapped to the input region is shown in FIG. 603, and the calculation method is:

[0076] hin(n) = (hout(n) * sh < cur_ph_abv(n))? 0 : hout(n) * sh - cur_ph_abv(n);

[0077] The calculated actual coordinate hin(n) of the H-direction of the current hblk input tensor data is one of the variables for the pooling input control device to read the input tensor data.

[0078] Step 3-1: Calculate the size of the bottom padding area of the current HBLK, cur_ph_bot(n); for the same reason, the area of the H direction block may be partially in the bottom padding area, so the original configuration of the ph_bot size needs to be adjusted. According to steps 1 and 2, the lower boundary coordinates of the current HBLK are calculated, denoted as:

[0079] cur_bot_coord(n) = hin(n) + hin_size_all;

[0080] Calculate the sum of the current cropping area H direction height and the top padding area, denoted as:

[0081] hin_add_ph_abv = hin_crop_len + ph_abv, where hin_crop_len and ph_abv are always configuration parameters.

[0082] According to cur_bot_coord(n) and hin_add_ph_abv, cur_ph_bot(n) can be obtained, denoted as:

[0083] cur_ph_bot(n) = (cur_bot_coord(n) <= hin_add_ph_abv)? 0:

[0084] cur_bot_coord(n) - hin_add_ph_abv;

[0085] Step 4: Get the H direction height of the current output HBLK mapped to the input HBLK, as shown in FIG. 604. Since there may be padding areas, the actual height of hin_size(n) may need to be adjusted, denoted as:

[0086] hin_size(n) = hin_size_all - cur_ph_abv(n) - cur_ph_bot(n);

[0087] The HBLK block data is the core processing unit of the present application, and the block information of the current HBLK is obtained by the above steps. The hardware device of the present application processes the hblk in the current cblk, and combines Figure 7A detailed explanation follows. As shown in Figure 701, in this embodiment, the pooling kernel is kw*kh. Using the block segmentation method described above, the first block input of HBLK is obtained as hblk_in(0). According to Figure 702, the current cblk segmentation size is defined as cblk_size, the data bit width of each feature point is d_size, and the depth of each single-port SRAM is sram_h. That is, each SRAM can cache the size hblk_size*cblk_size*d_size shown in Figure 703. Depending on the size of the pooling kernel kw, the output hout*c plane of each hblk_size height may be stored in multiple SRAMs. Figure 8 The SRAM shown will combine its SRAM mapping method with... Figure 8 A detailed explanation is provided.

[0088] Figure 8 This is a schematic diagram of the organization structure of temporary data in SRAM provided by the pooling device in an embodiment of the present invention. The temporary data storage method proposed in this invention uses the H direction as the block, that is, it stores the temporary data pooled in the H direction. However, it is also preferable to use the W direction as the block to store the temporary data in the W direction. Both methods are within the protection scope of this invention. Figure 8 The internal cache SRAM of the device of the present invention is dynamically adaptive, and the influencing factor is the range of the current pooling kernel size kw in the W direction of this pooling support [1, Kw_max]. The method of the present invention is as follows: Figure 8 To illustrate with an example, let's set the current pooling device's single-port SRAM size to sram_h = 64, cblk_size = 16, d_size = 8 bits, and the pooling device supports a maximum pooling size of [Kw_max, Kh_max] = [32, 32]. Then, we can see that this example requires 33 1Kbyte single-port SRAMs.

[0089] like Figure 8 The SRAM dynamically adaptively partitions into four modes, as shown in Figure 801. When kw ∈ [Kw_max / 2, Kw_max] (i.e., kw ∈ [16, 32]), HBLK obtains the current maximum block size hblk = sram_h = 64 based on the above. Each hin*cblk_size plane is determined according to... Figure 8 The device shown performs pooling in the H direction, saves the temporary results in each SRAM, and after the SRAM array has stored kw hout*cblk_size planes, it performs pooling in the W direction and outputs the final pooling result through the data output device.

[0090] As shown in FIG. 802, when kw∈[Kw_max / 4,Kw_max / 2] range (this embodiment also kw∈[8,16), sram hardware dynamically adapts to the array structure shown, HBLK block height size hblk_size=sram_h*2=128, reducing the read-in of input redundant data (as shown in FIG. 604, there is kh redundant input data read-in per block);

[0091] As shown in FIG. 803, when kw∈[Kw_max / 8,Kw_max / 4] range (this embodiment also kw∈[4,8), sram hardware dynamically adapts to the array structure shown, HBLK block height size hblk_size=sram_h*4=256;

[0092] As shown in FIG. 804, when kw∈[1,Kw_max / 8] range (this embodiment also kw∈[1,4), sram hardware dynamically adapts to the array structure shown, HBLK block height size hblk_size=sram_h*8=512;

[0093] As shown in the above embodiments, the mapping method of sram sets four modes 801-804, and the present application does not limit the specific kw segmentation range, and a range greater than 4 or less than 4 can also be set, and other embodiments adopted by those skilled in the art without other creative labor based on the method of the embodiments of the present application all belong to the scope of the present application.

[0094] It should be noted that the present embodiment sets cblk_size=16, and the present application device needs to set an operation device for each period operation cblk_size in the H direction and the W direction, but the unit of the operation unit of the present application device is not limited, and the calculation speed can also be reduced. However, in order to maximize the throughput efficiency of the pooling operation, the present application device needs to balance between the external input data bandwidth and the cblk_size*d_size of sram.

[0095] Figure 9 is a structure schematic diagram of the pooling device provided by the embodiment of the present application. The running process of the present application device is combined with the method adopted in the foregoing embodiments, which includes several parts:

[0096] The pooling top-level control device 901 is used to analyze the information of the current CMD in the present embodiment, and the overall control process between batch->cblk->hblk is carried out according to Figure 1 , and some top-level control information of other devices is provided;

[0097] Input / output control device 902, which is used to control the reading and writing back of the tensor feature data and the pooling result of the pooling core after the HBLK unit block information is determined, and its source can be other online modules or offline external storage medium;

[0098] HBLK unit control device 903, which is used for the flow control of the HBLK block in the embodiment, including controlling the pooling process in the H and W directions, selecting the maximum pooling value or the average value pooling process, and performing the pooling operation on the HBLK block according to the requirement of the top-level control device;

[0099] Pooling operation device 904, including the maximum value pooling core and the average value pooling, which contains the independent units in the H and W directions in each unit in the embodiment.

[0100] The above embodiment combines the specific implementation process of the pooling core size within the range of the device of the application, that is, kw·kh∈[Kw_max,Kh_max], and the following will explain kw·kh outside the range of [Kw_max,Kh_max], Figure 10 Figure 1001 illustrates the schematic diagram of the pooling core according to the CMD splitting calculation method provided by the embodiment of the application, which illustrates how the large kernel size pooling core is split into multiple commands CMD, which is used for the process suitable for the device of the application. The splitting method of the large kernel pooling has universality.

[0101] Figure 1001 illustrates the schematic diagram of the original image in the W / H direction before splitting, which illustrates the case that kw·kh>[Kw_max,Kh_max], and the pooling core is split into four regions according to the CMD splitting method of the application. The embodiment illustrates Kw_max<kw<=2*Kw_max, Kh_max<kh<=2*Kh_max, but is not limited to this splitting example. According to the correlation between kw / kh and the maximum pooling core size [Kw_max,Kh_max] supported by the device of the application, it is possible to split into two CMDs, six CMDs, etc., which all belong to the protection range of the large kernel pooling splitting method of the application. According to 1001, the current pooling step size parameter is sw, sh, the W / H cropping region length is win_crop_len / hin_crop_len, and the W / H starting position coordinates are win_st, hin_st. According to the method, it is split into four CMD0 (1002), CMD1 (1003), CMD2 (1004), and CDM3 (1005), and the pooling large kernel kw·kh is split into four pooling core blocks kw_0·kh_0, kw_1·kh_0, kw_0·kh_1, and kw_1·kh_1.

[0102] Figure 1002 is a CMD0 of a split of an upper left part (kw_0*kh_0) of the original pooling kernel (kw*kh) of Figure 1001, the split ranges of kw_0 and kh_0 satisfy the conditions of 1kw_0<=Kw_max, 1kh_0<=Kh_max, the top-level control configures the cropping area as win crop len0 / hin crop len0, and the starting position coordinates of W / H as win st0, hin st0. win crop len0=win crop len-(kw-kw_1), hin crop len0=hin crop len-(kh-kh_1), win st0=win st, and hin st0=hin st. CMM0 performs operations according to the overall process proposed in the foregoing of the present application, and outputs the pooling result of the split to an offline storage medium.

[0103] Figure 1003 is a CMD1 of a split of an upper right part (kw_1*kh_0) of the original pooling kernel (kw*kh) of Figure 1001, the split ranges of kw_1 and kh_0 satisfy the conditions of 1kw_1<=Kw_max, 1kh_0<=Kh_max, the top-level control configures the cropping area as win crop len1 / hin crop len0, and the starting position coordinates of W / H as win st1, hin st0. win crop len1=win crop len-(kw-kw_0), win st1=win st+kw_0, and CMM1 performs operations according to the overall process proposed in the foregoing of the present application, and outputs the pooling result of the split to an offline storage medium.

[0104] Figure 1004 is a CMD2 of a split of a lower left part (kw_0*kh_1) of the original pooling kernel (kw*kh) of Figure 1001, the split ranges of kw_0 and kh_1 satisfy the conditions of 1kw_0<=Kw_max, 1kh_1<=Kh_max, the top-level control configures the cropping area as win crop len0 / hin crop len1, and the starting position coordinates of W / H as win st0, hin st1. hin crop len1=hin crop len-(kh-kh_0), win st0=win st, and hin st1=hin st+kh_0. CMM0 performs operations according to the overall process proposed in the foregoing of the present application, and outputs the pooling result of the split to an offline storage medium.

[0105] Figure 1005 is a CMD3 of the original pooling kernel (kw·kh) left lower part (kw_1·kh_1) split, the kw_1 and kh_1 split range meets the condition of 1kw_1<=Kw_max, 1kh_1<=Kh_max, the top layer control configuration cropping area is win_crop_len1 / hin_crop_len1, the starting position coordinates of W / H is win_st1, hin_st1. The parameter values are defined above. The CMM3 operates according to the overall process proposed above in the application, and the pooling result of the subblock is output to the offline storage medium

[0106] Figure 1006 shows the final result of the split CMD embodiment using a vector module to perform offline pooling operation to obtain the final large kernel pooling structure. The offline vector module is in the CNN hardware device, generally using the maximum value, mean value operation unit in the elt-wise module, or also using CPU software to operate; this part belongs to the general content in the art, which is not described in the application.

[0107] The above describes some specific embodiments of the application, but the protection scope of the application is not limited thereto, any modification and replacement within the technical scope disclosed in the application should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

[0108] The above embodiments are only for illustrating the technical concept and characteristics of the application, the purpose is to enable those skilled in the art to understand the content of the application and to implement it, and cannot limit the protection scope of the application. Any equivalent changes or modifications made according to the essence of the application should be covered in the protection scope of the application.

Claims

1. A method of convolutional neural network large kernel pooling, used for AI chip pooling layer hardware circuit, characterized in that, The method comprises: The HBLK / WBLK block mode of the mapping of the pooling output to the input comprises: The operation of the pooling kernel kh / kw length feature data is performed on the H / W direction dimension, and the temporary operation result of the direction is saved in the internal cache SRAM, and the maximum output temporary data is stored in the form of data, and when the internal cache size range is exceeded, the H / W direction block of cblk is performed; W, H direction represents the width and height direction of the feature map; kw / kh is the size of the W / H direction pooling kernel; The HBLK / WBLK block mode of the mapping of the pooling output to the input further comprises: The range of the pooling kernel operation is trimmable, including trimming in the W, H, and C directions, and when the internal HBLK / WBLK control unit determines that blocking is needed according to the internal cache and the current pooling parameters, the HBLK / WBLK input unit information of the pooling operation device is automatically operated by the hardware device according to the output parameters and the current block dynamic coordinate information; The method further comprises: When the size of the pooling kernel is greater than the maximum range [Kw_max, Kh_max] supported by the hardware device, a command CMD multiple splitting mode is defined, the configuration information of each splitting kernel is configured, and the final pooling operation result is obtained by combining the partial kernel results after multiple operations, so as to support the pooling kernel with kw>Kw_max or kh>Kh_max; Kw_max / Kh_max is the maximum length of the pooling kernel supported by the current pooling hardware device in the W / H direction. The method further comprises: The pooling tensor feature data is stored externally, comprising: The tensor feature data is divided in the C direction dimension by cblk_size; the C direction dimension is the channel dimension.

2. The method for large kernel pooling of the convolutional neural network according to claim 1, wherein The data of the current cblk(n) is stored in the form of W·(H·csize), and in the cblk(n), the data is continuously stored in the form of h(m)·cblk_size in a W plane, and is continuously saved in the last W plane of the current cblk(n), and then the tensor data of the next cblk(n+1) block is stored; cblk(n) represents the block size of the C direction divided by cblk_size, and n>=1. 3.The method of pooling a convolutional neural network kernel according to claim 1 or 2, wherein, The cblk_size is a power of 2. 4.The method of pooling a convolutional neural network kernel according to claim 1 or 2, wherein, The method further comprises: The storage of the pooling operation temporary result in the internal cache comprises: The pooling kernel size parsed by the current CMD command is kw·kh, wherein kw / kh is the size of the W / H direction pooling kernel, and Kw_max / Kh_max is the maximum length of the pooling kernel supported by the current pooling hardware device in the W / H direction; Kw_max+1 single-port SRAMs are set as internal caches, wherein the data width of a single SRAM is defined as cblk_size*d_size, wherein d_size is the bit width of the pooled feature data, the depth of the SRAM is set as sram_h, and a single SRAM can store the intermediate result data of sram_h*cblk_size*d_size of a W plane in the current cblk.

5. The method of pooling a convolutional neural network kernel as recited in claim 4, wherein, The storage of the temporary result of the pooling operation in the internal cache further includes that the SRAM structure for storing the temporary data is dynamically adaptive, when kw is less than a set threshold, the sram in the H direction is increased in a combined manner to increase the depth of the temporary data, so as to reduce the effect of reading the redundant tensor feature data, and a larger pooling cblk block is operated.

6. An apparatus for CNN kernel pooling, configured to implement the method for CNN kernel pooling according to claim 1, wherein The device comprises: a pooling internal top-level control device configured to analyze command information configured from an external top-level control module, and to divide the internal device into a loop control flow of batch (n) -> cblk (m) -> hblk (k) according to the command information, wherein n, m, and k >= 1, and the hblk (k) block is the smallest core operation unit thereof; a pooling feature data input and output device configured to control reading or outputting of tensor data from an offline DRAM device or an online module, wherein the data block information is provided by the pooling internal top-level control device, and the data block size is one hblk unit; an HBLK unit control device configured to read input cache raw data, and to control the pooling device to perform an H direction -> W direction operation flow, and finally to output the pooling operation result to an output device; a pooling operation device comprising maximum value and average value operation circuits in H and W directions, respectively, wherein the operation circuits in each direction of the two pooling operations share the same set of register cache units, when the cache data is greater than kh / kw, the corresponding pooling operation is performed on the effective data, and the pooling result is output.

Images