Pooling Unit for Deep Learning Acceleration

Abstract

Various embodiments of the present disclosure relate to a pooling unit for deep learning acceleration. A convolutional neural network includes a pooling unit. The pooling unit performs pooling operations between convolutional layers of the convolutional neural network. The pooling unit comprises a hardware block that improves computational efficiency and area efficiency in the convolutional neural network.

Description

Technical Field

[0001] The present disclosure generally relates to convolutional neural networks implemented in electronic systems. Background Art

[0002] Deep learning algorithms have facilitated very high performance in many applications involving recognition, identification, and / or classification tasks. However, this progress can come at the expense of significant processing power requirements. Consequently, their adoption can be hampered by a lack of low-cost and energy-efficient solutions. Therefore, when deploying deep learning applications on embedded devices, stringent performance specifications can coexist with strict constraints on power consumption and energy consumption. Summary of the Invention

[0003] One embodiment is a pooling unit of a convolutional neural network. The pooling unit includes a cropper configured to receive a feature tensor and generate a cropped feature tensor comprising a plurality of data values by cropping the feature tensor. The pooling unit includes a line buffer configured to receive data values from the cropper; a column calculator configured to perform a column pooling operation on columns of data from the line buffer; and a row calculator configured to perform a row pooling operation on rows of data from the column calculator.

[0004] One embodiment is a method that includes receiving a feature tensor in a pooling unit of a convolutional neural network and generating a cropped feature tensor comprising a plurality of data values by cropping the feature tensor using a cropper of the pooling unit. The method includes passing the data values of the cropped feature tensor to a single-port row buffer of the pooling unit. The method includes generating pooled feature data by performing a column pooling calculation and a row pooling calculation on the data values from the row buffer.

[0005] One embodiment is a method. The method includes receiving a feature tensor in a pooling unit of a convolutional neural network, storing pooling window size data in a configuration register of the pooling unit, and generating a plurality of pooling windows from the feature tensor using the pooling unit based on the pooling window size data. The method includes generating pooled feature data by performing column pooling calculations and row pooling calculations on data values from the pooling windows. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram of an electronic device according to one embodiment.

[0007] Figure 2 is a block diagram of the processing flow within a convolutional neural network according to one embodiment.

[0008] Figure 3 is a representation of a feature tensor according to one embodiment.

[0009] Figure 4 is a block diagram of a pooling unit according to one embodiment.

[0010] Figure 5A is a schematic diagram of a cropping operation performed by a cropper according to one embodiment.

[0011] Figure 5B is a schematic diagram of a cropping operation of a cropper according to one embodiment.

[0012] Figure 6A is a schematic diagram of a row buffer of a pooling unit according to one embodiment.

[0013] Figure 6B The operation of a row buffer of a pooling unit according to one embodiment is shown.

[0014] Figure 7A The operation of the fill control of the pooling unit according to one embodiment is shown.

[0015] Figure 7B is a schematic diagram of a column calculator of a pooling unit according to one embodiment.

[0016] Figure 8 The operation of a batch buffer of a pooling unit according to one embodiment is shown.

[0017] Figure 9 The operation of the batch buffer and row calculator of the pooling unit according to one embodiment is shown.

[0018] Figure 10 A pooling operation performed by a pooling unit according to one embodiment is shown.

[0019] Figure 11 is a flowchart of a method for operating a convolutional neural network according to one embodiment.

[0020] Figure 12 is a flowchart of a method for operating a convolutional neural network according to one embodiment. DETAILED DESCRIPTION

[0021] Figure 1 1 is a block diagram of an electronic device 100 according to one embodiment. The electronic device 100 includes a convolutional neural network (CNN) 102. The CNN 102 receives input data 110 and generates prediction data 112 based on the input data 110. The CNN 102 generates the prediction data 112 by performing one or more convolution operations on the input data 110.

[0022] In one embodiment, input data 110 is provided by an image sensor (not shown) or another type of sensor of electronic device 100. Thus, input data 110 may include image data corresponding to one or more images captured by the image sensor. The image data is formatted so that it can be received by CNN 102. CNN 102 analyzes input data 110 and generates prediction data 112. Prediction data 112 indicates a prediction or classification related to one or more aspects of the image data. Prediction data 112 may correspond to a shape, object, face, or other aspect of the identified image. Although some embodiments herein describe receiving input data 110 from a sensor or sensor system, input data 110 may be received from other types of systems or devices without departing from the scope of this disclosure. For example, input data 110 may include a data structure that is stored in a memory and contains statistical data collected and stored by an external CPU. Other types of input data 110 may be used without departing from the scope of this disclosure.

[0023] In one embodiment, CNN 102 is trained through a machine learning process to recognize aspects of training images provided to CNN 102. The machine learning process includes passing a plurality of training images having known features to the CNN. The machine learning process trains CNN 102 to generate prediction data that accurately predicts or classifies the features of the training images. The training process may include a deep learning process.

[0024] CNN 102 includes multiple convolution units 104 and pooling units 106. Convolution units 104 implement the convolution layers of CNN 102. Therefore, each convolution unit is a hardware block that performs the convolution operation corresponding to the convolution layer. Pooling unit 106 implements the pooling function between convolution layers. Convolution units 104 and pooling units 106 work together to generate prediction data 112 from input data 110.

[0025] In one embodiment, each convolution unit 104 includes a convolution accelerator. Each convolution unit 104 performs a convolution operation on feature data provided to the convolution unit 104. Feature data is generated from input data 110. The convolution operation at the convolution layer convolves the feature data with kernel data generated during the machine learning process for CNN 102. The feature data generated by the convolution operation varies according to the kernel data. New feature data is provided from one convolution unit 104 to the next.

[0026] Pooling operations are performed on feature data between convolutional layers. When feature data is passed from one convolutional layer to the next, a pooling operation is performed on the feature data to prepare the feature data for the convolution operation of the next convolutional layer. Pooling unit 106 performs pooling operations between convolutional layers. Pooling unit 106 is used to accelerate convolutional neural network operations. Pooling unit 106 can perform maximum pooling operations, minimum pooling operations, average pooling operations, or other types of pooling operations.

[0027] The output of the convolutional layer is a tensor or a series of sensors. Tensors are similar to matrices in that they include multiple rows and columns with data values in various data fields. The pooling operation takes a portion of the feature tensor (such as a pooling window) and generates a pooled sub-tensor with a reduced dimension compared to the pooling operation. Each data field in the pooled sub-tensor is generated by performing a specific type of mathematical operation on multiple data fields from the feature tensor (such as taking the maximum, minimum, or average value from these data fields). The pooling operation is performed on each part of the feature tensor. The various pooled tensors are passed to the next convolutional layer as the feature tensor of that convolutional layer. Therefore, pooling helps to reduce the data used for the next convolution operation and arrange the data for the next convolution operation.

[0028] To keep it simple, Figure 1 CNN 102 of FIG1 only shows convolution unit 104 in pooling unit 106. However, in practice, CNN 102 may include many other hardware blocks, which may include batch normalization blocks, scaling blocks, bias blocks, normalization blocks, activation blocks, and other types of hardware blocks that perform various operations as part of CNN 102.

[0029] Figure 2 1 is a simplified block diagram of the processing flow within CNN 102 according to one embodiment. CNN 102 includes an input layer 120, multiple convolutional layers 105, and one or more fully connected layers 122. Input data 110 is provided to input layer 120 and passes through various convolutional layers 105 and fully connected layers 122. The output of the final fully connected layer is predicted data 112.

[0030] Each convolutional layer 105 performs a series of convolution operations on the input data 110 or on data generated from the input data 110 by the previous layer of the CNN 102. In particular, kernel data is associated with each convolutional layer 105. Each convolutional layer 105 performs a convolution operation between the kernel data of the convolutional layer 105 and feature data provided to the convolutional layer 105. The feature data is derived from the input data 110.

[0031] In one embodiment, the first convolutional layer 105 receives feature data from the input layer 120. The feature data of the first convolutional layer 105 is the input data 110. The first convolutional layer 105 generates feature data from the input data 110 by performing a convolution operation between the feature tensor of the input data 110 and the kernel tensors of the tensor data. The first convolutional layer then passes the feature data to the second convolutional layer 105. As used herein, each convolutional layer 105 receives feature data and convolves the feature data with the kernel data, and the output of the convolution operation is also referred to as feature data herein. Therefore, each convolutional layer 105 receives feature data and generates adjusted feature data based on the convolution operation. The adjusted feature data is then passed to the next convolutional layer 105, which further adjusts the feature data by performing a convolution operation on the feature data.

[0032] A pooling operation 107 is performed on the feature data between each convolutional layer. As described above, the pooling operation reduces the dimensionality of the feature data by performing arithmetic operations on the feature data. The pooling operation also prepares and organizes the feature data for the convolution operation of the next convolutional layer 105. The pooling unit 106 performs the pooling operation 107.

[0033] This convolution and pooling process repeats until the final convolutional layer 105 performs a convolution operation on the received feature data. A pooling operation 107 is performed on the feature data from the final convolutional layer 105. The feature data is then provided to a fully connected layer 122. The fully connected layer 122 then generates prediction data 112 from the feature data. In one embodiment, the pooling operation generates pooled feature data.

[0034] In practice, CNN 102 consists of Figure 2 . In particular, CNN 102 may include batch normalization operations, scaling operations, bias operations, normalization operations, activation operations, and other types of operations. Furthermore, the output of the pooling operation may be provided to processes, layers, modules, or components other than convolutional layers without departing from the scope of this disclosure.

[0035] Figure 3 is a representation of a feature tensor 128 according to one embodiment. The feature tensor 128 includes a plurality of blocks. Each of these blocks represents a data value. The tensor 128 includes a height, a width, and a depth. Although Figure 3 The feature tensor 128 is shown as a 5x5x5 tensor, but in practice, the feature tensor 128 may include other height, width, and depth dimensions.

[0036] In one embodiment, during the pooling operation, the feature tensor 128 is divided into batches. The feature tensor 120 is patched by depth. The pooling operation is performed on the batches from the feature tensor. The pooling operation is performed on the sub-tensors from each batch. Thus, each batch is divided into multiple sub-tensors.

[0037] Figure 4 1 is a block diagram of pooling unit 106 according to one embodiment. Pooling unit 106 is part of the hardware blocks that make up CNN 102. Pooling unit 106 includes stream filter 142, cropper 144, row buffer 146, padding control 148, column calculator 150, batch buffer 152, row calculator 154, multiplexer 156, average multiplier stage 158, stride manager 160, multiplexer 162, and non-pooling block 163. Input stream link 141 provides data to pooling unit 106. Pooling unit 106 outputs data to output stream link 165. Configuration register 164 stores configuration data for pooling unit 106. Pooling unit 106 performs pooling operations between convolutional layers of CNN 102.

[0038] The input to the pooling unit 106 is a 3D feature tensor 128 divided into sub-tensors called batches. The 3D feature tensor 128 flows into the pooling unit 106 via an input flow link 141. The width of the flow link 141 can be configured at design time, thereby configuring the maximum width of the data carried in the flow link 141.

[0039] In one embodiment, the pooling unit 106 includes a stream filter 142 at its input. The stream filter receives the feature tensor data 128. The stream filter 142 ensures that only validated streams are input to the pooling unit 106. In one embodiment, the stream filter 142 can be augmented with a first-in-first-out (FIFO) buffer to buffer input data from the stream link 141.

[0040] In one embodiment, the feature tensor 128 is read into the pooling unit 106 from the stream link 141, first by depth, then by width from left to right, and finally by height from top to bottom. Thus, in one embodiment, the data is read in a classic raster zigzag scan order across width and height.

[0041] In one embodiment, the cropper 144 is configured to crop the input feature tensor 128. In some cases, the pooling operation may be performed only on certain portions of the feature tensor 128. The cropper 144 can be configured to crop the feature tensor 128 to extract only the portion of the feature tensor 128 on which the pooling operation is to be performed.

[0042] Figure 5A1 is a diagram of a cropping operation performed by the cropper 144 according to one embodiment. The feature tensor 128 is input to the cropper 144. The cropper 144 generates a cropped feature tensor 168 from the input feature tensor 128.

[0043] The user may select the dimensions of the cropped feature tensor 168 via the configuration registers 164. In particular, data may be written to the configuration registers 164 that instruct the cropper 144 on how the input feature tensor 128 should be cropped. Thus, the configuration registers include dedicated configuration data for the cropper 144. The configuration registers may be set with a height index, a width index, and a depth index for cropping the feature tensor 128.

[0044] Figure 5B FIG. 1 is a diagram illustrating a cropping operation of the cropper 144 according to one embodiment. The feature tensor 128 is represented by indices 1 a to 9 c , each index representing a data position within the feature tensor 128 . Figure 5B Also shown is the result of performing a clipping operation on the feature tensor 128.

[0045] Figure 5B Also included are flattened representations of the feature tensor 128 and the cropped feature tensor 168. The flattened representation represents the three-dimensional feature tensor 128 and the cropped feature tensor 168 as two-dimensional tensors.

[0046] Figure 6A According to one embodiment Figure 4 14. The row buffer 146 receives an input stream from the cropper 144. The input stream from the cropper 144 corresponds to the cropped feature tensor 168. Alternatively, if the cropper 144 is not configured to crop the feature tensor 128, the input stream from the cropper 144 may be the feature tensor 128. Figure 6A The input stream shown in corresponds to Figure 5B The index of the pruned feature tensor 168.

[0047] The row buffer 146 is organized from the internal memory of the pooling unit 106. The row buffer 146 buffers the input horizontal rows in the memory. The row buffer 146 can extract and output vertical columns from the stored rows upon request. In addition, the row buffer 146 allows the reuse of previously buffered rows when storing new incoming rows.

[0048] In one embodiment, the row buffer 146 receives an input stream and outputs feature data in feature data columns 170. Each feature data column 170 includes data values from a particular position in each row of the row buffer 146. For example, the first column includes data values from the first position in each row of the row buffer 146. The second column includes data values from the second position in each row of the row buffer 146. The third column corresponds to data values from the third position in each row of the row buffer 146, and so on.

[0049] In one embodiment, the number of rows and the width of the rows define the size of the memory for row buffer 146. For example, a row buffer with five rows can buffer output vertical columns up to 5 in height.

[0050] Figure 6B 1 shows the operation of the line buffer 146 of the pooling unit 106 according to one embodiment. In particular, Figure 6B 14 shows the order in which new rows are filled in the row buffer 146 when all existing rows are full. Figure 6B In the example of FIG, a new row of data is received at row buffer 146. The top row of the row buffer in its current state will be replaced with the new row of data. Thus, the new buffer state includes the new row of data in the top row of row buffer 146.

[0051] In one embodiment, row buffer 146 is a single-port row buffer. This means that row buffer 146 includes a single input port and a single output port. As a result, single-port row buffer 146 has very low power consumption and occupies a small area on the integrated circuit. This is very beneficial in terms of power consumption, area consumption, and general efficiency of pooling unit 106.

[0052] Figure 7A The operation of the fill control 148 of the pooling unit 106 is shown according to one embodiment. The fill controller 148 receives the feature data column 170 from the line buffer 146. The fill controller 148 may fill the feature data column 170 by input padding provided by a user.

[0053] In some cases, padding may be required to adjust the size of the input data window to the desired pooling window because strides are made on the input. This may mean that additional feature data columns 170 and / or one or more additional rows are required for each data column. In this case, the user can configure the configuration data in the configuration register 164 so that the padding control 148 pads the feature data columns 170.

[0054] exist Figure 7AIn the example shown, the height of feature data column 170 is 2. Padding control 148 is configured to add an extra row of zeros and an extra column of zeros. Thus, padding control 148 generates padded feature data column 172. Padded feature data column 172 includes an extra column of zeros and an extra row of zeros in each column. Other padding configurations may be selected without departing from the scope of this disclosure. Padding control 148 may also be configured to not perform any padding. In this case, the output of padding control 148 is the same as the input to padding control 148.

[0055] Figure 7B is a schematic diagram of a column calculator 150 of the pooling unit 106 according to one embodiment. The column calculator 150 receives padded feature data columns 172 from the padding control 148. The column calculator 150 generates output data 174, which includes, for each column, a corresponding data value.

[0056] Output data 174 from column calculator 150 is based on the type of operation selected for column calculator 150. Configuration data in configuration register 164 may define the type of operation to be performed by column calculator 150. Some examples of the types of operations that a column calculator may perform include maximum operations, minimum operations, and average operations.

[0057] Figure 7B Two types of output data 174 are shown. The top output data corresponds to a maximum operation performed by the column calculator 150. In the maximum operation, the column calculator determines the maximum value of the populated feature data column 172 for each input. The column calculator 150 outputs a data value corresponding to the maximum value of the data values in the column for each input populated feature data column 172. Figure 7B In the example of , the maximum value of the padded feature data column 172 of the first input is 6, the maximum value of the padded feature data column 172 of the second input is 4, and the maximum value of the padded feature data column 172 of the third input is 2. Therefore, the output data 174 of the maximum operation are 6, 4, and 2.

[0058] The bottom output data 174 corresponds to the summation calculation of the column calculator 150. For the summation calculation, the column calculator 150 generates the sum of the data values in the input padded feature data column 172 for each input padded feature data column 172. The sum of the data values in the padded feature data column 172 of the first input is 11. The sum of the data values in the padded feature data column 172 of the second input is 7. The sum of the data values in the padded feature data column 172 of the third input is 3. Therefore, the output data 174 of the summation operation includes the data values 11, 7, and 3.

[0059] Figure 8The operation of the batch buffer 152 of the pooling unit 106 according to one embodiment is shown. The batch buffer 152 receives output data 174 from the column calculator 150. The batch buffer 152 stores the output data 174 in rows and columns. The batch buffer 152 receives the output data 174 via the column demultiplexer 176. The batch buffer consists of a plurality of rows and columns. The maximum number of rows corresponds to the maximum batch size that the unit is designed to support. For example, if the maximum batch size is 8, the number of rows is 8, so the unit supports tensors with batch sizes ranging from 1 to 8. In this specific example, the tensor batch size is 3, so 3 rows are occupied, even though the actual number of rows may be larger. Similarly, the unit can be designed to have a number of columns corresponding to the maximum pooling window width dimension expected to be supported. In this figure, the number of columns is 3, so the unit can support pooling window widths of 1, 2, or 3.

[0060] The column demultiplexer 176 receives the output output data 174 and places it into the desired batch buffer column based on the batch index associated with the incoming column. In this example, the data value OP(1a, 1b, 1c) corresponds to batch index 0 (batch size = 3, so the index range is 0, 1, 2) and is therefore placed in the first batch buffer row. Similarly, the next data value OP(2a, 2b, 2c) corresponds to batch index 1 and is therefore placed in the second batch buffer row, and so on, while the data value OP(4a, 4b, 4c) corresponds to batch index 0 and is therefore placed in batch buffer row 0, but column index 1 because it is the result of the second column along the width dimension of the feature tensor. Therefore, in Figure 8 In the example of , OP(1a, 1b, 1c) corresponds to output data 174 generated by performing the selected operation of the column calculator on padded feature data column 172, which includes indices 1a, 1b, and 1c corresponding to from feature tensor 128 or pruned feature tensor 168. Column demultiplexer 176 outputs the various data values in data 174 in rows and columns of batch buffer 152 in a selected manner.

[0061] Figure 9 The operation of the batch buffer 152 and the row calculator 154 of the pooling unit 106 is shown according to one embodiment. The batch buffer 152 outputs rows of output data 174 to the row multiplexer 178. The row calculator 154 receives rows of data from the row multiplexer 178.

[0062] In one embodiment, row calculator 154 performs an operation on each row from batch buffer 152. Row calculator 154 outputs a data value for each row corresponding to the operation performed on the row. The type of operation performed by row calculator 154 is stored in configuration register 164. These operations may include a maximum operation, a minimum operation, or a sum operation.

[0063] Figure 10 1 shows a pooling operation performed by the pooling unit 106 according to one embodiment. The configuration register 164 determines the size of the pooling window used for each pooling operation. The pooling window corresponds to a selected portion of the cropped feature tensor 168. The size of the pooling window determines, in part, the magnitude of the size reduction between the input to the pooling unit 106 and the output of the pooling unit 106.

[0064] exist Figure 10 In the example shown, the pooling window is 3x3. If the height and width of the cropped feature tensor 168 are 9x9, the pooling window selected is 3x3, and the stride is 3 (described in more detail below), then the pooling operation will generate a single data value for each non-overlapping 3x3 window of a single depth slice of the cropped feature tensor 168. This corresponds to nine data values from each 9x9 slice of the cropped feature tensor 168.

[0065] exist Figure 10 In the example of , a 3x3 sub-tensor is selected from the cropped feature tensor 168. The 3x3 sub-tensor is passed to the column calculator 150. The column calculator 150 is configured to provide, for each column, the sum of the data values in that column. Thus, the output of the column calculator is the data values 12, 15, and 18. The row calculator 154 is also configured to perform a sum operation. Thus, the row calculator 154 generates the data value 45, which is the sum of 12, 15, and 18.

[0066] In one example, the pooling operation selected is averaging rather than summing. In this case, the column calculator 150 and the row calculator 154 each generate a sum, such as Figure 10 . The sum 45 is then provided to the average multiplier stage 158. The average multiplier stage 158 performs an average operation on the sums from the row calculator 154. The output of the average multiplier stage is a value of 5. 5 is the average of the values in the 3x3 pooling window. This operation is performed for all pooling windows until an average value is generated for each pooling window. A reduced feature tensor has been generated by the pooling unit 106, which includes each of the values generated for each pooling window in the pooling window.

[0067] In one embodiment, average multiplier stage 158 includes a multiplier 180 and a regulator unit 182. The effect of average multiplier stage 158 is to divide the value provided by row calculator 154 by the number of data values in the pooling window. Figure 10In the example of , the effect of the average multiplier stage 158 is to divide the output of the row calculator 154 by 9. However, division operations are very expensive in terms of computing power. Therefore, the multiplier converts the fraction 1 / 9 into its fixed point Q15 form 3641. The multiplier then multiplies 3641 by 45, resulting in a value of 163845. This number is provided to the regulator unit. The regulator unit 182 downshifts, rounds, and saturates (if so configured) the value to a value of 5, which is the average of the values in the pooling window. In this way, the average multiplier stage 158 can calculate the average of the pooling window.

[0068] The average multiplier stage 158 includes configurable downshifting or supports truncation, rounding, and saturation of downshifted values based on the value specified in the pooling unit configuration register 164. Various rounding modes can be supported, such as round to nearest value, round away from zero, round to nearest even number, etc. If desired, the output can be saturated by enabling saturation via the configuration register 164. If saturation is not enabled, the output will simply be truncated to output the data. Return to Figure 4 The stride manager 160 is responsible for gating the inputs and only allowing those sample values to pass based on the horizontal and vertical strides specified in the pooling. One embodiment may use two counters, one counting to the horizontal stride value and the other to the vertical stride value. The counters are triggered on each incoming input, and the outputs are gated unless the counters are all zero, thus allowing only those values that comply with the stride requirements to be output.

[0069] In one embodiment, the pooling unit 106 also supports global pooling. In particular, the pooling unit 106 includes a global pooling unit 166. The global pooling unit provides a dedicated data path to perform global pooling on tensors of arbitrary size. Global pooling differs from window pooling in that the pooling window contains the entire width and height of the input feature tensor. Global pooling bypasses the row buffer 146, padding control 148, column calculator 150, row calculator 154, and the batch buffer 152 that can be clocked, thereby saving runtime power. The global pooling unit 166 is also batch aware.

[0070] The pooling unit 106 also supports non-pooling. In particular, the non-pooling block 163 implements the non-pooling operation. The non-pooling operation expands the maximum and minimum pooling in the convolutional neural network topology through its application, especially in semantic segmentation. In maximum non-pooling, the non-pooling block 163 records the position of the maximum value in the pooling window for each pooling window. The feature tensor 128 or the pruned feature tensor 168 can be partially regenerated by the non-pooling operation. In particular, the maximum value is put back into the corresponding data field of these maximum values from the feature tensor 128 or the pruned feature tensor 168. Other data fields can be filled with zeros. In addition to putting the minimum value back into their data field and filling other data fields with zeros, the non-pooling block 163 can do the same thing for minimum pooling.

[0071] Figure 11 is a flow chart of a method 1100 for operating a convolutional neural network according to one embodiment. At 1102, the method 1100 includes receiving a feature tensor in a pooling unit of the convolutional neural network. At 1104, the method 1100 includes generating a cropped feature tensor comprising a plurality of data values by cropping the feature tensor using a cropper of the pooling unit. At 1106, the method 1100 includes passing the data values of the cropped feature tensor to a single-port row buffer of the pooling unit. At 1108, the method 1100 includes generating pooled feature data by performing a column pooling calculation and a row pooling calculation on the data values from the row buffer. At 1110, the method 1100 includes outputting the pooled feature data to a convolutional layer of the convolutional neural network.

[0072] Although method 1100 describes outputting the pooled feature data to a convolutional layer, without departing from the scope of the present disclosure, the pooled feature data may be output to a layer, process, component, or module other than a convolutional layer. Other variations of method 1200 may be made without departing from the scope of the present disclosure.

[0073] Figure 12 is a flow chart of a method 1200 for operating a convolutional neural network according to one embodiment. At 1202, the method 1200 includes receiving a feature tensor in a pooling unit of the convolutional neural network. At 1204, the method 1200 includes storing pooling window size data in a configuration register of the pooling unit. At 1206, the method 1200 includes generating, using the pooling unit, a plurality of pooling windows from the feature tensor according to the pooling window size data. At 1208, the method 1200 includes generating pooled feature data by performing a column pooling calculation and a row pooling calculation on data values from the pooling windows. At 1210, the method 1200 includes outputting the pooled feature data to a convolutional layer of the convolutional neural network.

[0074] Although method 1200 describes outputting the pooled feature data to a convolutional layer, without departing from the scope of the present disclosure, the pooled feature data may be output to a layer, process, component, or module other than a convolutional layer. Other variations of method 1200 may be made without departing from the scope of the present disclosure.

[0075] Further details related to electronic devices that implement convolutional neural networks can be found in U.S. Patent Application Publication 2019 / 0266479, filed on February 20, 2019, U.S. Patent Application Publication 2019 / 0266485, filed on February 20, 2019, and U.S. Patent Application Publication 2019 / 0266784, filed on February 20, 2019, each of which is incorporated herein by reference in its entirety.

[0076] The various embodiments described above can be combined to provide further embodiments. These and other changes can be made to the embodiments in light of the above detailed description. Generally, in the appended claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and claims, but should be construed to include all possible embodiments and the full scope of equivalents to which such claims are entitled. Therefore, the claims are not limited by this disclosure.

Claims

1. A pooling unit of a convolutional neural network, comprising: a clipper configured to receive a feature tensor and generate a clipped feature tensor comprising a plurality of data values by clipping the feature tensor; a line buffer configured to receive the data value from the cropper; a column calculator configured to perform a column pooling operation on the data columns from the row buffer; a row calculator configured to perform a row pooling operation on the data rows from the column calculator; and A configuration register is configured to store configuration data for the cropper, the column calculator, and the row calculator. 2 . The pooling unit of claim 1 , further comprising a global pooling data path configured to selectively bypass the row buffer, the column calculator, and the row calculator. 3 . The pooling unit according to claim 1 , further comprising a non-pooling block configured to perform a non-pooling operation on the pooled data from the row calculator and the column calculator. 4 . The pooling unit of claim 1 , wherein the configuration register stores data indicating a type of pooling operation to be performed by the column calculator and the row calculator.

5. The pooling unit of claim 1 , wherein the configuration register stores data indicating how the cropper should crop the feature tensor. 6 . The pooling unit of claim 1 , further comprising an average multiplication stage configured to generate an average value from the row pooled data generated by the row calculator. 7 . The pooling unit of claim 1 , further comprising a fill control block configured to fill the data column from the row buffer and pass the filled data column to the column calculator. 8 . The pooling unit of claim 1 , wherein the row buffer is a single-port row buffer, wherein the single-port row buffer comprises a single input port and a single output port.

9. The pooling unit of claim 1 , further comprising a stride manager configured to control a window stride when pooling the cropped feature tensor.

10. A method for operating a convolutional neural network, comprising: Receive feature tensors in the pooling unit of the convolutional neural network; generating a cropped feature tensor comprising a plurality of data values by cropping the feature tensor using a cropper of the pooling unit; passing the data values of the cropped feature tensor to a single-port row buffer of the pooling unit, wherein the single-port row buffer includes a single input port and a single output port; generating pooled feature data by performing a column pooling calculation and a row pooling calculation on the data values from the row buffer; as well as Configuration data for the clipper is stored in a configuration register of the pooling unit.

11. The method according to claim 10, further comprising: The convolutional neural network is utilized to generate prediction data based at least in part on the pooled feature data. 12 . The method of claim 10 , wherein the configuration register stores data indicating a type of pooling operation to be performed by a column calculator and a row calculator of the pooling unit.

13. The method of claim 10, wherein the configuration register stores data indicating how the cropper should crop the feature tensor. The method of claim 10 , wherein the configuration data stores pooling window size data indicating a size of a pooling window.

15. The method according to claim 10, further comprising: The pooled feature data is output to the convolutional layer of the convolutional neural network.

16. A method for operating a convolutional neural network, comprising: Receive feature tensors in the pooling unit of the convolutional neural network; Storing pooling window size data in a configuration register of the pooling unit; generating a cropped feature tensor comprising a plurality of data values by cropping the feature tensor with a cropper of the pooling unit; generating a plurality of pooling windows from the feature tensor using the pooling unit according to the pooling window size data; Generating pooled feature data by performing column pooling calculations and row pooling calculations on data values from the pooling window; as well as Configuration data for the clipper is stored in a configuration register of the pooling unit.

17. The method according to claim 16, further comprising: A non-pooling block of the pooling unit is used to perform a non-pooling operation on the pooled feature data.

18. The method according to claim 16, further comprising: The convolutional neural network is utilized to generate prediction data based at least in part on the pooled feature data.

19. The method of claim 16, further comprising: The pooled feature data is output to the convolutional layer of the convolutional neural network.

Images