Pooling computation implementation system

Abstract

The application discloses a system for implementing a pooling calculation, which comprises a horizontal pooling module and a vertical pooling module, wherein the horizontal pooling module performs a horizontal pooling operation on an input feature map and outputs the feature map after size arrangement; and the vertical pooling module temporarily stores the feature map after the horizontal pooling operation and performs a vertical pooling operation. The application has a high flexibility and a general pooling operation architecture, and can adapt to different sizes of the pooling operation in various cases while effectively saving the area consumption.

Description

Technical Field

[0001] This invention relates to a technology in the field of neural networks, specifically a pooling computation implementation system. Background Technology

[0002] Pooling is one of the main computational methods in convolutional neural networks. Current neural network operations are primarily implemented using CPUs or GPUs, but the limitations of their hardware architecture restrict the potential for improvement. Customized hardware platforms such as ASICs and FPGAs offer advantages in terms of latency and power consumption. While some current research on ASICs and FPGAs supports pooling operations, it cannot adapt to the diverse pooling computation methods used in the wide variety of neural networks. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes a pooling computing implementation system. Through a highly flexible and universal pooling operation architecture, it can adapt to pooling operations of different sizes under various conditions while effectively saving area consumption.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a pooling computation implementation system, comprising: a horizontal pooling module and a vertical pooling module, wherein: the horizontal pooling module performs horizontal pooling operations on the input feature map and outputs the data after resizing; the vertical pooling module temporarily stores the feature map after horizontal pooling and performs vertical pooling operations.

[0006] The horizontal pooling module includes a horizontal pooling tree generation unit and a data processing unit, wherein: the horizontal pooling tree generation unit performs the maximum value operation in max pooling or the sum operation in average pooling on any row of the received feature map; the data processing unit resizes the feature map output by the horizontal pooling tree generation unit and outputs it to the vertical pooling module.

[0007] The vertical pooling module includes a storage array and a pooling operation unit. The storage array temporarily stores the feature map output by the horizontal pooling module through several storage sub-units. The pooling operation unit performs vertical pooling processing based on the feature map output by the horizontal pooling module and the previously stored feature map. When the pooling task is not completed, the result feature map is output; otherwise, the vertical pooling result is stored in the storage array.

[0008] Technical effect

[0009] This invention provides a hardware structure for pooling designs of different sizes, which can support multiple pooling structures and reduce the area of ​​the hardware. Attached Figure Description

[0010] Figure 1 Schematic diagram of the hardware structure of the pooling module;

[0011] Figure 2 Schematic diagram of the structure of the maximum pooling tree;

[0012] Figure 3 is a schematic diagram of the processing process of the pooling tree adapting to different sizes;

[0013] Figure 4 Schematic diagram of the average pooling tree;

[0014] Figure 5 Schematic diagram of the vertical pooling calculation process. Specific implementation manner

[0015] As Figure 1 shown, a pooling calculation implementation system involved in this embodiment includes: a horizontal pooling module and a vertical pooling module, where: the feature map is input into the horizontal pooling module, and after being processed by the horizontal pooling tree generation unit and the data arrangement unit, it is output to the vertical pooling module; the storage array in the vertical pooling module stores the feature map data to be vertically pooled, and the pooling operation unit performs a pooling operation on the data output from the horizontal pooling module and the data temporarily stored in the storage array at each moment, and determines whether to store the calculation result in the storage array or output it as the final result.

[0016] The horizontal pooling module described above includes: a horizontal pooling tree generation unit and a data arrangement unit, where: the horizontal pooling tree generation unit performs a maximum value extraction operation in maximum pooling or an addition operation in average pooling on any row of the received feature map; the data arrangement unit re-integrates the size of the feature map output by the horizontal pooling tree generation unit and then outputs it to the vertical pooling module.

[0017] As Figure 2 shown, the horizontal pooling tree generation unit generates a pooling tree including several tree-like structural processing nodes according to the input feature map, and performs a horizontal pooling operation on each row of the input feature map data. When the width of the top-level feature map meets the pooling size condition, zero-padding processing is performed on both sides of the feature map.

[0018] The pooling size condition refers to: when it satisfies that the width of the feature map before pooling is L1 < S×(L₂ - 1) + N, zero-padding processing is required, where: the width of the feature map after pooling is L₂, the pooling size is N×N, and the pooling stride is S.

[0019] As Figure 2As shown, the first layer of the pooling tree is a row of the input feature map. Zeros are padded to both ends of the feature map during input. Then, adjacent data in this row are compared pairwise at the processing nodes to find the maximum value. This result serves as the input to the second layer of the pooling tree, and so on, constructing a multi-layered pooling tree. In the example shown, the results of the second and third layers (2x2 size, 2x2 stride maximum pooling and 3x3 size, 2x2 stride maximum pooling) are contained within the second and third layers of the pooling tree. Since the stride is 2, only one value from the second and third layers needs to be taken, resulting in the outputs of the 2x2 size, 2x2 stride maximum pooling and the 3x3 size, 2x2 stride maximum pooling, respectively.

[0020] The diagram only shows examples of two sizes. In reality, max pooling of any size can be extracted from the results of the corresponding layer, demonstrating the high flexibility and versatility of this structure. Since feature map size transformations are common in neural networks, a fixed-size pooling tree typically needs to be adapted to handle feature maps of various sizes, as detailed below:

[0021] As shown in Figure 3, when the width of the feature map input to the horizontal pooling module is 8 pixels, and the corresponding pooling tree processes 8 pixels at a time, the pooling tree can process the feature map data of one output channel simultaneously in each clock cycle. When the width of the input feature map is 4, the same pooling tree can process the feature map data of two output channels simultaneously. When the horizontal pooling processing parameter is 2*2, since the number of feature map columns is exactly divisible by 2, no special processing is required for adaptation. When the horizontal pooling processing parameter is 3*3, since the number of feature map columns is not divisible by 3, ... Figure 3b As shown, the pixels transmitted by the path indicated by the dashed arrow will affect the processing of the feature map data of the second output channel, requiring feature processing for adaptation.

[0022] The adaptation refers to the following: the horizontal pooling tree generation unit controls the validity of the pixels transmitted by the dashed arrow through a mask. Specifically, when the mask is closed, i.e. the pixel is valid, the feature map data is transmitted; otherwise, 0 is transmitted. By invalidating unnecessary duplicate paths, i.e. setting them to zero, duplicate comparisons or pixel additions are avoided.

[0023] like Figure 4 As shown, the processing node is a comparator or an adder. When a comparator is used, the pooled number is subjected to maximum pooling. When an adder is used, the pooled number is subjected to average pooling.

[0024] The vertical pooling module includes a storage array and a pooling operation unit. The storage array temporarily stores the feature map output by the horizontal pooling module through several storage sub-units. The pooling operation unit performs vertical pooling processing based on the feature map output by the horizontal pooling module and the previously stored feature map. When the pooling task is not completed, the result feature map is output; otherwise, the vertical pooling result is stored in the storage array.

[0025] like Figure 5 As shown, taking a memory array containing four memory sub-cells as an example, the feature map has L1 in the first row, L2 in the second row, L3 in the third row, and so on; in the figure This represents the comparison operation in max pooling or the addition operation in average pooling. From time T1 to T4, the first to fourth storage sub-units RAM1-RAM4 are sequentially allowed to store data from rows L1 to L4 of the feature map. Specifically: when L1 arrives at time T1, L1 is stored in RAM1; when L2 arrives at time T2, the data in RAM1 is read out and processed together with L2, the result is stored back in RAM1, and L2 is stored in RAM2, and so on. The arriving data is processed and stored in the manner described in the diagram.

[0026] Based on the number of storage sub-cells N, the data storage time of each storage sub-cell is N clock cycles. That is, after every N-1 rows of data, the current storage sub-cell will be cleared and the data of the next moment will be stored. The example in the figure has a maximum size of N×N.

[0027] The pooling operation unit reads the corresponding storage sub-units at different times according to the results of different vertical pooling sizes, and then performs comparison or addition operations. This allows the number of storage sub-units to be reduced according to size requirements. Specifically, as shown in the figure, the second storage sub-unit RAM2 and the fourth storage sub-unit RAM4 within the dashed box are not used in the example sizes, so they can be reduced in this embodiment.

[0028] When more sizes need to be supported, simply use a similar method to retrieve the data or expand the storage sub-units.

[0029] Through specific practical experiments, the pooling module was implemented using SMIC 40NM ASIC technology. The network latency and hardware area consumed by the module were tested using the ordinary pooling module and the present invention in three networks: VGG16, ResNet, and RepVGG.

[0030] Table 1

[0031]

[0032]

[0033] Compared with the prior art, the present invention can support more types of pooling in terms of the number of pooling operations. On the other hand, as shown in the table, under the condition of having the same latency as ordinary pooling, the hardware area of ​​the pooling module of the present invention is reduced by 12.9% compared with ordinary pooling, while implementing the same number and types of pooling.

[0034] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A pooling computing implementation system, characterized in that, It includes: A horizontal pooling module and a vertical pooling module, where: the feature map is input into the horizontal pooling module, and after being processed by the horizontal pooling tree generation unit and the data arrangement unit, it is output to the vertical pooling module; the storage array in the vertical pooling module stores the feature map data to be vertically pooled, and the pooling operation unit performs a pooling operation on the data output from the horizontal pooling module and the data temporarily stored in the storage array at each moment, and determines whether to store the calculation result in the storage array or output it as the final result; The described horizontal pooling module includes: a horizontal pooling tree generation unit and a data arrangement unit, where: the horizontal pooling tree generation unit performs a maximum value operation in maximum pooling or an addition operation in average pooling on any row of the received feature map; the data arrangement unit re-integrates the size of the feature map output by the horizontal pooling tree generation unit and then outputs it to the vertical pooling module; The described horizontal pooling tree generation unit generates a pooling tree including several tree-like structural processing nodes according to the input feature map, and performs a horizontal pooling operation on each row of the input feature map data. When the width of the top-level feature map meets the pooling size condition, zero-padding processing is performed on both sides of the feature map; The described pooling size condition means that when it is satisfied that the width of the feature map before pooling is L1 < S×(L2 - 1) + N, zero-padding processing is required, where: the width of the feature map after pooling is L2, the pooling size is N×N, and the pooling stride is S.

2. The pooling computing implementation system according to claim 1, characterized in that, When the number of columns of the feature map cannot be divided evenly by the horizontal pooling processing parameters, feature processing needs to be performed for adaptation. Specifically: the horizontal pooling tree generation unit controls the validity of the pixels transmitted by the dotted arrow through the mask. Specifically: when the mask is closed, that is, when the pixels are valid, the data of the feature map is transmitted, otherwise 0 is transmitted. By invalidating the unnecessary repeated addition paths, that is, setting them to zero, the repeated addition of pixels is avoided.

3. The pooling computing implementation system according to claim 1, characterized in that, The described processing node is a comparator or an adder. When a comparator is used, the pooling number performs maximum pooling processing. When an adder is used, the pooling number performs average pooling processing.

4. The pooling computing implementation system according to claim 1, characterized in that, The described vertical pooling module includes: a storage array and a pooling operation unit, where: the storage array temporarily stores the feature map output by the horizontal pooling module through several storage subunits, and the pooling operation unit performs vertical pooling processing according to the feature map output by the horizontal pooling module and the previously temporarily stored feature map. When the pooling task is completed, the result feature map is output, otherwise the vertical pooling result is continued to be stored in the storage array.

5. The pooling computing implementation system according to claim 4, characterized in that, Based on the number N of storage subunits, the data storage time of each storage subunit is N clock cycles, that is, after every N - 1 rows of data, the current storage subunit will be cleared and store the data of the next moment. The example in the figure shows up to the size N×N at most.

6. The pooling computing implementation system according to claim 4, characterized in that, The described pooling operation unit performs a comparison operation or an addition operation after reading the corresponding storage subunits at different moments according to the results of different vertical pooling sizes, so as to reduce the number of storage subunits according to the size requirements.

Images