A neural network pooling layer acceleration method

Abstract

The present disclosure belongs to the technical field of neural networks, and particularly relates to a neural network pooling layer acceleration method, which comprises the following steps: obtaining configuration parameter information; obtaining information data of a tensor block according to the configuration parameter information; the tensor block comprises an input tensor block and an output tensor block, and the information data comprises a cache address of the tensor block; obtaining input data based on the cache address; performing width direction pooling on the input data according to the configuration parameter information, and caching process data; performing height direction pooling based on the process data; and repeating the pooling step until all input tensor blocks are traversed. The time consumption of the pooling operation is shortened.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of neural networks, and in particular relates to a neural network pooling layer acceleration method. BACKGROUND

[0002] In the field of artificial intelligence, convolutional neural networks have been widely used in recent years. The basic structure is shown in the structural diagram as Figure 6 shown, which is composed of an input layer, a hidden layer and an output layer. The output feature data of the previous layer is used as the input feature of the next layer, and the data is continuously transmitted backward until the final output of the network is obtained. The hidden layer between the input layer and the output layer can be further divided into linear layers such as convolutional layer, pooling layer and fully connected layer, and nonlinear layers (also known as activation layer) such as ReLU and Sigmoid according to different application scenarios.

[0003] Early hardware implementation methods are difficult to support larger size pooling windows, and usually need to perform multiple rounds of smaller size pooling operations to realize larger size pooling operators. This implementation method will cause the data interaction between the off-chip storage device and the pooling acceleration unit to exceed several times the original data amount, causing communication bandwidth tension. When it is impossible to complete through small size pooling, algorithm developers need to modify the pooling operator parameters while ensuring algorithm accuracy, which brings certain pressure to algorithm personnel. SUMMARY

[0004] The present disclosure is proposed based on the above needs of the prior art, and the technical problem to be solved by the present disclosure is to provide a neural network pooling layer acceleration method to shorten the time-consuming of pooling operation.

[0005] To solve the above problems, the technical scheme provided by the present disclosure comprises:

[0006] A neural network pooling layer acceleration method is provided, comprising: obtaining configuration parameter information; obtaining information data of a tensor block according to the configuration parameter information, the tensor block comprising an input tensor block and an output tensor block, and the information data comprising a cache address of the tensor block; obtaining input data based on the cache address; performing pooling in the width direction on the input data according to the configuration parameter information, and caching process data; performing pooling in the height direction based on the process data; repeating the pooling step until all input tensor blocks are traversed.

[0007] Preferably, the configuration parameter information comprises basic information of the pooling layer, and related information of input tensor blocks and output tensor blocks; the basic information of the pooling layer comprises a pooling type, a pooling window height, a pooling window width, a pooling height direction step, a pooling width direction step, input data height direction boundary zero padding information, and input data width direction boundary zero padding information; the related information of the input tensor blocks comprises an input block position serial number, an input tensor block height, an input tensor block width, and an input channel number; and the related information of the output tensor blocks comprises an output tensor block height and an output tensor block width.

[0008] Preferably, the storage mode of the tensor block comprises splitting into two dimensions, PDI and T, along a channel direction; data is stored along the channel direction; when data storage along the channel direction reaches a PDI boundary, data is stored along a width direction; when data storage along the width direction reaches a width boundary, data is stored along a height direction; and when data storage along the height direction reaches a height boundary, data is stored along the channel direction to a high dimension T.

[0009] Preferably, the information data of the tensor block obtained according to the configuration parameter information comprises obtaining a height index of an input tensor block element based on the parameter information and information of an output tensor block element, the information of the output tensor block element comprising position information of the output tensor block element obtained by four-dimensional traversal of the output tensor block, and the height index of the input tensor block element being represented as: wherein n tih is the height index of the input tensor block element, n toh is the height index of the output tensor block element, S h is a pooling height direction step, k h is a pooling window height, P h is input data height direction boundary zero padding information, max tih is a maximum value of the height index of the input tensor block element.

[0010] Preferably, the information data of the tensor block obtained according to the configuration parameter information further comprises obtaining a width index of an input tensor block element based on the parameter information and information of an output tensor block element, the information of the output tensor block element comprising position information of the output tensor block element obtained by four-dimensional traversal of the output tensor block, and the width index of the input tensor block element being represented as: wherein n tiw is the width index of the input tensor block element, n tow is the width index of the output tensor block element, S w is a pooling width direction step, k w is a pooling window width, P wmax tiw The maximum value of the width index of the input tensor block element.

[0011] Preferably, the cache address of the tensor block is calculated by the following expression: A i = n if × N ih × N iw + n ih × N iw + n iw wherein n if is the channel index, n ih and n iw are the coordinates of the feature element in the feature map, N ih is the height of the input feature map, and N iw is the width of the input feature map.

[0012] Preferably, the pooling layer obtains PDI data from the external storage module each time; the pooling layer comprises PDI poolers, and the PDI poolers are parallel pooled.

[0013] Preferably, the pooling layer outputs a feature matrix represented as: T(n of , n oh , n ow ) = func(X(n if , n ih : n ih + k h , n iw : n iw + k w )) wherein Y() is the output feature matrix output by the pooling layer, the position of the Y matrix output element in the feature map is (n oh , n ow ), the channel index is n of , func() represents Max() for maximum pooling, func() represents Average() for average pooling, X() is the input feature matrix in the pooling window, the position of the upper left element of the X matrix in the feature map is (n ih , n iw ), the channel index is n if , k h is the height of the input feature matrix in the pooling window, and k w is the width of the input feature matrix in the pooling window.

[0014] Compared with the prior art, the present disclosure proposes an acceleration method for the pooling layer of the convolutional neural network according to the characteristics of the pooling operation in the CNN. With the smallest FPGA resource consumption, the pooling operation supports any pooling sliding step, pooling kernel size and the number of boundary zero padding. The implementation reduces the number of times of reading input data from the external storage module, reduces the data communication amount and data access overhead, thus shortens the time consumption of the pooling operation, thereby reducing the operation cost of the pooling operation and improving the efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present specification, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0016] Figure 1 A step flow chart of the acceleration method for the neural network pooling layer provided by the present application;

[0017] Figure 2 A storage mode diagram of the tensor block in the embodiment of the present application;

[0018] Figure 3 A feature map block diagram in the embodiment of the present application;

[0019] Figure 4 A pooling along the width direction diagram in the embodiment of the present application;

[0020] Figure 5 A pooling along the height direction diagram in the embodiment of the present application;

[0021] Figure 6 A basic structure diagram of the convolutional neural network in the prior art. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0023] In the description of the embodiments of this disclosure, it should be noted that, unless otherwise expressly specified and limited, the term "connected" should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.

[0024] Throughout the text, the terms “top,” “bottom,” “above,” “below,” and “on top” refer to the relative positions of components of the device, such as the relative positions of the top and bottom substrates within the device. It is understood that the device is multifunctional and independent of its spatial orientation.

[0025] To facilitate understanding of the embodiments of this application, the following will provide further explanation and description with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this application.

[0026] This embodiment provides a method for accelerating neural network pooling layers, such as... Figures 1-5 As shown.

[0027] like Figure 1 As shown, the neural network pooling layer acceleration method includes:

[0028] Get configuration parameter information.

[0029] Typically, the input data for a pooling layer is the feature data output by the convolutional layer of a neural network. In other words, for a pooling acceleration unit, its input is the input feature data, and the default sorting method is fmin[H][W][C], where C is the channel direction, W is the width direction, and H is the height direction. The channel direction has the lowest dimension, the height direction has the highest dimension, and the width direction has a dimension in between the two.

[0030] like Figure 2 As shown, the channel direction C is split into two dimensions: one part is placed in the low-dimensional PDI to achieve parallel channel processing, and the other part is placed in the high-dimensional T to achieve data partitioning, which facilitates batch operation of the pooling layer. The storage method of the tensor block includes: data is stored along the channel direction; when the data storage in the channel direction reaches the PDI boundary, it is stored along the width direction; when the data storage in the width direction reaches the width boundary, it is stored along the height direction; when the data storage in the height direction reaches the height boundary, it is stored along the channel direction towards the higher dimension T, and the sorting method becomes fmin[T][H][W][PDI].

[0031] The configuration parameter information includes basic information about the pooling layer, as well as relevant information about the input tensor block and the output tensor block; the basic information about the pooling layer includes the pooling type (Pool).mode , pooling window height K h , pooling window width K w , pooling height direction step size S h , pooling width direction step size S w , input data height direction boundary zero padding information P h , and input data width direction boundary zero padding information P w ; the related information of the input feature tensor block includes input block position serial number Tensor block height direction index is idx h , Tensor block channel direction index is idx c . Input tensor block height H i , input tensor block width W i , input channel number N i ; the related information of the output feature tensor block includes output tensor block height H o , output tensor block width W o .

[0032] Due to the limited on-chip storage resources, all input feature map data cannot be loaded. Therefore, a block division method is adopted for calculation, and the specific block division method is shown in Figure 3 . The feature map is divided along the height direction and the channel direction, and each block is a 3-dimensional tensor block. The number of height direction and channel direction blocks is TN h , TN c . Since the pooling layer usually has the requirement of zero padding at the input feature map boundary, according to the position of the Tensor in the feature map, the Tensor is divided into three categories, which have the same channel number and width but different heights. The first category of Tensor has idx h equal to 0, and the height of the Tensor affected by the boundary zero padding is less than the default height. The second category of Tensor has idx h greater than 0 and less than TN h , which is a default size tensor block. The third category of Tensor has idx h equal to TN h -1, which is affected by the input feature map size and the boundary zero padding, and its height is usually smaller.

[0033] According to the configuration parameter information, information data of a tensor block is obtained; the tensor block includes an input tensor block and a cache tensor block, and the information data includes a cache address of the tensor block.

[0034] Specifically, according to the obtained configuration parameter information, the height, width and channel number of each input tensor block Tensor_Proc in are calculated and output, and the height, width and channel number of each output tensor block Tensor_Proc outheight, width and channel number of the input tensor. The input tensor block is cached in the input tensor cache module, and the corresponding process tensor is cached in the process tensor cache module, and the address and control signal of the input tensor cache module and the process tensor cache module are generated, and the input feature and the process data are sent to the pooling layer for pooling operation.

[0035] The whole processing flow is from the output Tensor out The four-dimensional traversal is PDI, H, W and T respectively. According to the output element position information, the input information participating in the calculation is deduced, and the input element height n tih and width index n tiw can be obtained by the following expression, which is specifically obtained by transforming the output element height index n toh and width index n tow .

[0036]

[0037]

[0038] When the input data index calculated exceeds the Tensor in input feature map, the address A i of the input feature element participating in the calculation of the flow is obtained by the following expression:

[0039] A i = n if × N ih × N iw + n ih × N iw + n iw

[0040] Wherein, n f is the channel index, n ih and n iw are the coordinates of the feature element in the feature map, N ih is the height of the input feature map, and N iw is the width of the input feature map.

[0041] Based on the cache address, the input data is obtained; the address of the input tensor block is obtained according to the above steps, and then the input data is obtained.

[0042] As shown in Figure 4 and Figure 5 , the input data is pooled in the width direction according to the configuration parameter information, and the process data is cached; the process data is pooled in the height direction; repeat the pooling step until all input tensor blocks are traversed.

[0043] Since the channel direction is independent when performing pooling, there is no data dependency, so the embodiment selects parallel calculation along the channel direction to improve the calculation performance. The pooling layer includes PDI pooling units, and each pooling unit is responsible for the pooling operation of the feature map of one channel.

[0044] Generally, K w *K h -1 operation is required to complete 1 pooling operation. For maximum pooling, K w *K h -1 comparison operation is required to obtain one output element. For average pooling processing, K w *K h -1 addition and 1 division operation is required to obtain one output element. Wherein, K w is the width of the pooling kernel, and K h is the height of the pooling kernel.

[0045] To reduce the calculation amount, the embodiment introduces a process tensor cache module to decompose the pooling operation into two steps of width direction pooling and height direction pooling. Through decomposition, the results of width direction pooling can be fully utilized, and the calculation complexity can be greatly reduced from O(K w *K h ) to O(K w +K h ), wherein O() is the complexity calculation. At the same time, the reading frequency of the original data can be greatly reduced. K w *K h times of original data reading are required to complete one pooling operation early, and now only K w times of original data reading and K h times of original data reading are required, reducing the dependence on data communication bandwidth.

[0046] Specifically, the pooling process includes:

[0047] Width direction pooling, reading K w data X(n tif , n tih , n tiw ) from the input tensor cache module, performing 1xK w pooling operation in a pipeline, and writing the result data to the process tensor cache module until all rows are traversed.

[0048] For maximum pooling, the operation is as follows:

[0049] max(reg if , X(n if , n tih , n tiw))

[0050] pooling register reg if The initial value of the register reg is the minimum value of the current representable range, for example, the minimum value corresponding to 8-bit width is -128. When the index value of an element in X(n if , n ti , n tiw ) is invalid, the element value is also set to the minimum value.

[0051] For average pooling, the operation performed is as follows:

[0052] add(reg if , X(n if , n tih , n tiw ))

[0053] The initial value of the register reg if is 0, and when the index value of an element in X(n if , n tih , n tiw ) is invalid, the element is also set to 0.

[0054] In the height direction, the Tensor tmp is pooled in the height direction. r The pooling operation is K r ×1, K r is the number of data, and after K of data are read and operated, the final calculation data is output, and the process is repeated until all columns are traversed.

[0055] The output feature matrix of the pooling layer is represented as:

[0056] Y(n of , n oh , n ow ) = func(X(n if , n ih : n ih +k h , n iw : n iw +k w ))

[0057] where Y() is the output feature matrix output by the pooling layer, the position of the Y matrix output element in the feature map is (n oh , n ow ), the channel index is n of , for maximum pooling, func() represents Max(), for average pooling, func() represents Average(), X() is the input feature matrix in the pooling window, and the position of the upper left element of the X matrix in the feature map is (n ih, n iw ) the channel index is n if , k h is the height of the input feature matrix within the pooling window, k w is the width of the input feature matrix within the pooling window.

[0058] Based on the above discussion, a specific embodiment will be described here to facilitate understanding:

[0059] Assume that the size of the input feature map of the pooling acceleration unit is 64x100x100, i.e., 64 channels, width and height are both 100; the size of the pooling window is 10x10; the pooling step is 1; the boundary zero padding width is 9. The channel parallelism PDI of the pooling acceleration unit is 32. The input feature fmin[2]

[100]

[100]

[32] is stored in the external storage module in the manner.

[0060] Due to the limitation of on-chip cache, assume that the maximum storage of the process tensor cache module is a tensor block Tensor of 32x20x100, i.e., the number of channels is 32, the height is 20, and the width is 100. The height of the first type of Tensor is 20; the height of the second type of Tensor is 29; and the height of the third type of Tensor is 20.

[0061] First, obtain the configuration parameter information, calculate the specific information of each tensor block Tensor according to these information, and obtain the address and control signal of the input tensor cache module and the process tensor cache module.

[0062] Second, the input tensor cache module obtains the input tensor block from the external storage module and provides the data required for calculation to the pooling layer.

[0063] Third, the pooling layer performs a width direction pooling operation on the input feature data according to the configuration information, the pooling window size is 1x10, and the process data is stored in the process tensor cache module.

[0064] Fourth, the pooling layer obtains data from the process tensor cache module according to the configuration information, performs a height direction pooling operation, the pooling window size is 10x1, and outputs the final pooling result.

[0065] Fifth, repeat the third step to the fourth step until all input tensor blocks are traversed. At this point, the pooling operation is completed.

[0066] The application discloses a neural network pooling layer acceleration method which can support any pooling type, any step and any pooling window, and the working mode of the method can be configured by external input and interacted with other processing in the pooling layer.

[0067] The above detailed description is further detailed for the purpose of the application, technical solutions and beneficial effects, and it should be understood that the above description is only a specific embodiment of the application and is not used to limit the protection scope of the application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application should be included in the protection scope of the application.

Claims

1. A method for accelerating a neural network pooling layer, the method comprising: The method comprises: obtaining configuration parameter information; obtaining information data of a tensor block according to the configuration parameter information, the tensor block comprising an input tensor block and an output tensor block, the information data comprising a cache address of the tensor block; wherein the obtaining of the information data of the tensor block according to the configuration parameter information comprises obtaining a height index of an input tensor block element based on the parameter information and information of an output tensor block element, the information of the output tensor block element comprising position information of the output tensor block element obtained by traversing four dimensions of the output tensor block, and the height index of the input tensor block element being represented as: wherein, n tih is the height index of the input tensor block element, n toh is the height index of the output tensor block element, S h is the pooling height direction step, k h is the pooling window height, P h is the input data height direction boundary padding information, max tih is the maximum value of the height index of the input tensor block element; and a width index of the input tensor block element is obtained based on the parameter information and information of the output tensor block element, the information of the output tensor block element including position information of the output tensor block element obtained by traversing four dimensions of the output tensor block, and the width index of the input tensor block element is represented as: wherein n tiw is the width index of the input tensor block element, n tow is the width index of the output tensor block element, S w is the pooling width direction stride, k w is the pooling window width, P w is the input data width direction boundary padding information, max tiw is the maximum value of the width index of the input tensor block element; obtaining input data based on the cache address; performing pooling in a width direction on the input data according to the configuration parameter information, and caching process data; performing pooling in a height direction based on the process data; repeating the pooling step until all input tensor blocks are traversed.

2. The method according to claim 1, wherein: the configuration parameter information comprises basic information of a pooling layer, and related information of the input tensor block and the output tensor block; the basic information of the pooling layer comprises a pooling type, a pooling window height, a pooling window width, a pooling height direction step, a pooling width direction step, input data height direction boundary zero padding information, and input data width direction boundary zero padding information; the input tensor block related information comprises an input block position serial number, an input tensor block height, an input tensor block width, and an input channel number; the output tensor block related information comprises an output tensor block height and an output tensor block width.

3. The method according to claim 1, wherein: the storage mode of the tensor block comprises: splitting along a channel direction into two dimensions, i.e. a low dimension PDI and a high dimension T; storing data along the channel direction; when the data storage along the channel direction reaches a PDI boundary, storing data along a width direction; when the data storage along the width direction reaches a width boundary, storing data along a height direction; when the data storage along the height direction reaches a height boundary, storing data along the channel direction to the high dimension T.

4. The method according to claim 3, wherein: the cache address of the tensor block is calculated by the following expression: A i = n if x N ih x N iw + n ih x N iw + n iw where n if is the channel index, n ih and n iw are the coordinates of the feature element in the feature map, N ih is the height of the input feature map, and N iw is the width of the input feature map.

5. The method according to claim 1, wherein: the pooling layer obtains PDI data from an external storage module each time; the pooling layer comprises PDI poolers, and the PDI poolers perform parallel pooling.

6. The method according to claim 1, wherein: the output feature matrix of the pooling layer is represented as: Y(n of ,n oh ,n ow ) = func(X(n if ,n ih :n ih +k h ,n iw :n iw +k w )) Where Y() is the output feature matrix of the pooling layer, and the position of the output element of the Y matrix within the feature map is (n oh n ow The channel index is n of For max pooling, func() represents Max(); for average pooling, func() represents Average(). X() is the input feature matrix within the pooling window, and the position of the top-left element of the X matrix within the feature map is (n...). ih n iw The channel index is n if k h k is the height of the input feature matrix within the pooling window. w This is the width of the input feature matrix within the pooling window.

Images