Hardware-accelerated optimization group: Convolutional-based neural network model

Abstract

The present invention provides a method, system, and apparatus (including a computer-readable medium) for processing an input image using an integrated circuit that implements a convolutional neural network with group convolutional layers. [Solution] The process includes: determining the mapping of partitions along the channel dimension of the input feature map to multiplication-accumulation cells (MACs) in the circuit's computing device; applying group convolution to the input feature map; for each partition, providing weights for the group convolution layer to a subset of MACs based on the mapping; providing the input of the feature map to each MAC in the subset via the circuit's input bus; calculating the product at each MAC in the subset using the input and the weights for the group convolution layer; and generating the output feature map of the group convolution layer based on the accumulation of the products.

Description

Background Art

[0001] Background This specification generally relates to using a hardware integrated circuit to perform group convolutions of convolutional neural networks.

[0002] A neural network is a machine learning model that uses one or more node layers to generate an output such as a classification for a received input. Some neural networks include one or more hidden layers in addition to an output layer. Some neural networks can be convolutional neural networks configured for image processing or recurrent neural networks (RNNs) configured for audio and language processing. Various types of neural network architectures can be used to perform various tasks related to classification or pattern recognition, prediction involving data modeling, and clustering of information.

Summary of the Invention

[0003] A neural network layer can have a corresponding set of parameters or weights. The weights are used to process an input (e.g., a batch of inputs) through the neural network layer and generate a corresponding output of the layer for computing neural network inference. A batch of inputs and a set of kernels can be represented as tensors of inputs and weights, i.e., multi-dimensional arrays. A hardware accelerator is an application-specific integrated circuit for implementing a neural network. The circuit includes memory having positions corresponding to elements of tensors that can be traversed or accessed using the control logic of the circuit.

[0004] Summary This specification describes techniques for efficiently implementing group convolution on hardware neural network accelerators. Group convolution involves convolution of input feature maps by grouping them along the channel dimension of the input matrix, with each input group representing a group convolution associated with a corresponding output group. In particular, based on these techniques, leveraging group convolution to process input images using a convolutional neural network (CNN) of a machine learning model implemented on an exemplary computing device such as a tablet or smartphone can achieve certain hardware and computing efficiencies.

[0005] The input image is processed using a hardware integrated circuit that implements a convolutional neural network with group convolutional layers. The processing involves determining the mapping of partitions along the channel dimension of the input feature map to multiplication accumulator cells in the computing device of the integrated circuit, and applying group convolution to the input feature map. Applying group convolution involves, for each partition, providing the weights of the group convolutional layer to a subset of MACs based on the mapping, providing the input of the feature map to each MAC in the subset via the circuit's input bus, and, at each MAC in the subset, calculating the product using the input and the corresponding weights of the group convolutional layer. Based on the accumulation of the products, the output feature map of the group convolutional layer is generated.

[0006] One aspect of the subject matter described herein can be embodied in a method for processing an input image using a hardware integrated circuit configured to implement a convolutional neural network comprising multiple neural network layers. The method includes a group convolutional layer. This method includes identifying control parameters that define multiple partitions along the channel dimension of the input feature map, determining the mapping of the partitions to multiplication-accumulation cells (MACs) in the computing device of the integrated circuit, and, for the group convolutional layer, applying group convolution to the input feature map.

[0007] The application includes, for each partition, providing weights for a group convolutional layer to a subset of multiple MACs based on a determined mapping; providing the respective inputs of the input feature map to each MAC in the subset via the input bus of the integrated circuit; and, for each MAC in the subset, calculating the product using the respective inputs and the corresponding weights of the group convolutional layer. This method includes generating the output feature map of the group convolutional layer based on the accumulation of the products.

[0008] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, determining the mapping of partitions to multiplicative accumulation cells involves determining the mapping based on the number of channels in each partition. In some implementations, each partition of multiple partitions contains a corresponding number of input channels corresponding to the size of each partition.

[0009] Generating an output feature map involves generating an output feature map based on the respective size of each partition. In some implementations, generating an output feature map involves calculating multiple products using a subset of MACs and generating an accumulation of products from these multiple products. This method may include accessing information describing the hardware configuration of the computing device and determining the respective size of each partition based on the hardware configuration of the computing device.

[0010] In some implementations, the input bus includes a broadcast function, and the method further includes broadcasting multiple inputs of the input feature map to the integrated circuit's computing device for each partition via the input bus. This method may also include broadcasting the first input of the first partition of the input feature map to each MAC in the subset via the input bus, and the broadcasted first input is reused during the computation of the group convolutional layer. In some implementations, the first partition of the input feature map corresponds to the first partition of the output feature map, and the first input has reuse for the output of the first partition of the output feature map.

[0011] Other implementations of this and other embodiments include corresponding systems, devices, and computer programs configured to perform actions of the Method and encoded in computer storage devices. One or more computer systems may be configured so by software, firmware, hardware, or a combination thereof installed on the system, and the system performs actions during computation. One or more computer programs may be configured so, when executed by a data processing device, by having instructions that cause the device to perform actions.

[0012] The subject matter described herein can be implemented in specific embodiments to achieve one or more of the following advantages. We describe techniques for improving the performance of convolutional neural networks, including group convolutional layers, i.e., layers that perform group convolutions rather than depth-unit or full convolutions, by leveraging examples of dedicated integrated circuit hardware architectures.

[0013] Hardware architectures include specific types of memory layouts, broadcasting This includes an input bus and a configuration of multiplicative cells that can implement group convolutions with improved computational efficiency and hardware utilization compared to conventional architectures. The input bus is coupled to the multiplicative cells and configured to broadcast inputs to some (or all) of the multiplicative cells. The broadcasting feature allows for parallelization of input calculations that are reused when computing the output channels of the corresponding group convolutions.

[0014] This architecture can be used to optimize the execution of various types of group convolution-based neural networks, enabling the application of a broader range of group convolution concepts to various computer vision tasks. For example, a compiler or associated control logic can be used to determine the optimal mapping of group convolution operations to multiplication-accumulation cells within the circuit's computing device.

[0015] Mapping may be determined to optimize various aspects of computation, such as maximizing the overall utilization of computing devices, minimizing the overall latency of the computation, or both. A particular advantage of mapping is that it minimizes the number of off-chip communications required to obtain new or additional parameters for a given computation. The exemplary device determining the mapping (e.g., a host) may be off-chip relative to the integrated circuit. In some implementations, the compiler and other related control logic may be integrated into the exemplary device.

[0016] Details of one or more implementations of the subject matter described herein are given in the accompanying drawings and the following description. Other potential features, embodiments, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0017] [Figure 1] This is a block diagram of an exemplary computing system for performing group convolution on an image. [Figure 2] A block diagram illustrating groupings used in group convolution. [Figure 3] This figure shows exemplary attributes of machine learning models for different convolution operations. [Figure 4] This is a block diagram showing operations corresponding to different layer blocks of a convolutional neural network. [Figure 5] Figure 1 shows an exemplary architecture of a convolutional neural network model that can be used in the exemplary computing system. [Figure 6] This figure shows an exemplary hardware computing tile of a hardware integrated circuit used to perform computations in a convolutional neural network. [Figure 7A] A block diagram illustrating an exemplary mapping of partitions to a subset of multiplicative cumulative cells. [Figure 7B] This block diagram shows an exemplary input bus that provides inputs to the multiplication and accumulation cells of a hardware computation tile. [Figure 8] This is an illustrative block diagram illustrating certain attributes of full convolution, depth-unit convolution, and group convolution. [Figure 9] This is a diagram illustrating an exemplary process for applying group convolution using a hardware integrated circuit. [Modes for carrying out the invention]

[0018] Similar reference numbers and names in various drawings indicate the same elements. Detailed explanation Figure 1 shows an exemplary computer for performing group convolution on an input image. FIG. 0 is a block diagram of an imaging system 100. The system 100 generally includes an exemplary convolutional neural network 102 configured to process an image 104, i.e., to process the intensity values of the pixels of the image. The convolutional neural network 102 includes an exemplary neural network architecture based on a plurality of convolutional neural network layers 108. In the example of FIG. 1, the convolutional neural network 102 includes a plurality of convolutional neural network layers 108. For example, the convolutional neural network 102 includes N (or N sets) of layers. Here, N is an integer greater than 1.

[0019] Various types of CNN architectures 106 can be used to perform various machine learning tasks. For example, the machine learning task can be a computer vision task (also referred to as an “image processing task”). In other words, the neural network can be configured to receive an input image, process the input image to generate a network output of the input image, i.e., to perform some image processing task. As used herein, processing an input image refers to processing the intensity values of the pixels of the image using a neural network. For example, the task can be image classification, and the output generated by the neural network for a given image can be a score for each of a set of target categories, each score representing an estimate of the likelihood that the image contains an object of that category.

[0020] As another example, the task can be image embedding generation, and the output generated by the neural network can be a numerical embedding of the input image. As yet another example, the task can be object detection, and the output generated by the neural network can identify the position within the input image, e.g., a bounding box or other geometric region within an image in which a particular type of object is depicted. As yet another example, the task can be image segmentation, and the output generated by the neural network can define, for each pixel of the input image, to which of a plurality of categories that pixel belongs. However, more generally, the task can be any of a variety of tasks, including tasks that process inputs other than images.

[0021] Among image processing tasks, there can be associations with object detection, data classification, pattern recognition, or image recognition, and further with computational predictions involving data modeling or information clustering. For example, the task can include object detection, where the CNN processes an image to detect a particular object and generates an output that identifies the object upon detection. Another task can include data / image classification. In this case, the CNN processes an image to determine the classification of the image and generates a particular classification output for the image based on its content. Another task can include pattern recognition. In pattern recognition, the CNN processes an image to identify or recognize a particular pattern within the image and generates an output indicating the recognized pattern based on the content of the image. Another task can include general image recognition. In this case, the CNN processes an image to identify or recognize various elements of the image and generates an output indicating the recognized elements based on the content of the image.

[0022] In some implementations, the convolutional neural network 102 is implemented in or accessible by an exemplary mobile device 110. The mobile device 110 may be a smartphone, tablet, e-notebook, laptop, game console, or related portable computing device. In some other implementations, the convolutional neural network 102 is integrated into or accessible by an exemplary cloud-based system, such as a server bank, server group, or multiprocessor system.

[0023] The convolutional neural network 102 uses one or more machine learning hardware components. This can be implemented using an accelerator 112. Each hardware accelerator 112 corresponds to one or more dedicated hardware integrated circuits 114. Generally, the circuits 114 are hardware circuits (e.g., dedicated hardware circuits) that perform neural network computations. For example, some (or all) of the circuits 114 may be dedicated hardware circuits such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), single-core neural network processors, or multi-core neural network processors. The circuits 114 may also be dedicated graphics processing units (GPUs).

[0024] The hardware circuit 114 is computationally capable of accelerating the computation of the neural network workload. In some implementations, the hardware circuit 114 includes control logic, which can be implemented in hardware, software, or both. The control logic is used to issue instructions for the neural network computation, including the acquisition and routing of data used for computation. The circuit 114 may include memory for storing the inputs, input activations, outputs, output activations, and parameters of each layer of the neural network. In some implementations, the circuit 114 includes dedicated memory, shared memory, or both. For example, the circuit 114 may include input / activation memory for storing inputs, input activations, outputs, or output activations, and parameter memory for storing the respective parameter sets of each neural network layer.

[0025] Circuit 114 may include computing devices such as hardware matrix devices, arrangements of computation tiles, or combinations thereof. These computing devices are used to perform neural network computations to process inputs through layers of the neural network. In some implementations, each matrix device or individual computation tile contains one or more arrays of computation cells, such as multiplicative-accumulator cells that perform multiplication and accumulation operations. For example, each cell may multiply the input and weight values ​​to produce a product, and then accumulate the product (e.g., an addition operation) over multiple clock cycles.

[0026] Circuit 114 implements full convolution, depth-unit convolution, and group convolution to convolve filters with different weights on corresponding parts of the input matrix for a given depth of the channel dimension of the input matrix. For example, mobile device 110 uses a convolutional neural network 102 and a CNN layer 108 of the model to generate an image processing output 120, such as a recognition output or detection output for a received input 104. For example, input 104 could be an image of a laptop 122, and mobile device 110 uses the convolutional neural network 102 to process the image and detect or recognize that the image contains a depiction of the laptop.

[0027] Figure 2 is a block diagram including a representation of the input dataset 202 and an exemplary grouping 203 for performing group convolution using inputs from the input dataset. In some implementations, the input dataset 202 is or is derived from a multidimensional matrix structure of inputs. For example, the matrix structure may be an input tensor containing Zin channels, each with spatial dimensions X × Y. The matrix structure (or tensor) can represent either a set of inputs, a set of activation inputs, or a set of weight inputs. In some cases, this specification refers to the matrix structure of a set of activation inputs as an input feature map and the matrix structure of a set of weight inputs as a kernel matrix structure.

[0028] In the example in Figure 2, the input dataset 202 is a matrix structure (or tensor) with three dimensions: two (X, Y) spatial dimensions and one (Z) channel dimension. Regarding the spatial dimensions, in some implementations, these dimensions correspond to the space or position of the set of activation inputs. For example, if the convolutional neural network 102 is processing a two-dimensional image 104... In this case, the matrix structure can have two spatial dimensions corresponding to the spatial coordinates of the image, i.e., the X and Y coordinates. Regarding the channel dimension, this dimension corresponds to features from the input (e.g., the activation input). The channel dimension is described by reference to Z, Zin, or channel dimension, where "channel" may correspond to the color channels of the image.

[0029] System 100 is configured to determine the partitioning of group convolutions by referring, for example, to the depth levels of the channel dimensions of the input dataset 202. Each input channel may have a corresponding depth level. For example, the matrix structure in Figure 2 has depth levels that extend along the Zin dimension. As an example, if the exemplary matrix structure 202 represents a 3x3x3 image sent as a set of activation inputs to a convolutional neural network layer, then the X and Y dimensions (3x3) of the image may be spatial dimensions, and the Z dimension (3) may be the channel dimension corresponding to the R, G, and B values.

[0030] As described above, system 100 can determine the partitions of the group convolution along the channel dimension of the exemplary input feature map. For example, system 100 can determine a first partition of input group 210-1 along the channel dimension and a second partition of input group 210-2 along the channel dimension. In some implementations, system 100 determines n groupings 210-n along the channel dimension, where n is an integer greater than or equal to 1. In an example where the input feature map 202 represents a 3x3x3 image sent as a set of activation inputs, the first partition defining input group 210-1 of the group convolution may correspond to nine features of "1" activation inputs (e.g., red values), the second partition defining input group 210-2 of the group convolution may correspond to nine features of "2" activation inputs (e.g., green values), and the third partition defining input group 210-3 of the group convolution may correspond to nine features of "3" activation inputs (e.g., blue values).

[0031] As explained above, group convolution is performed by grouping input feature maps along the channel dimension of the input matrix, and each input group 210-n representing the group convolution is associated with a corresponding output group 220-n. The convolutional neural network 102 uses one or more convolutional neural network layers 108 to generate an output 206 (e.g., classification) for the received input 202. For example, each convolutional neural network layer is associated with a set of kernels 204. The kernels 204 can be partitioned according to the configuration of the group convolution so that each input group 210-n is convolved with the corresponding kernel / weight matrix to produce a convolutional output 220-n. In the example in Figure 2, input group 210-1 is convolved with the corresponding kernel matrix 212 to produce a convolutional output 220-1, while input group 210-2 is convolved with the corresponding kernel matrix 214 to produce a convolutional output 220-2.

[0032] System 100 is configured to dynamically determine the value of a control parameter g, where g is an integer greater than 1. System 100 is also configured to determine the group size by computing Zin / g, where Zin is the number of input channels along the channel dimension of the input tensor, and g is the number of groups defined by the control parameter. The control parameter g is used to define the number of group convolutions (e.g., partitions). In some cases, the value of g may be dynamically determined by System 100 or predefined by System 100 for a given operation. For example, the control parameter g defining the number of group convolutions may be predefined (and / or embedded) by the compiler of System 100 or determined dynamically at runtime.

[0033] In some implementations, system 100 defines the number of group convolutions (e.g., splits) based on the specific type of machine learning task being requested, and sets the value of the control parameter g according to that task. In some other implementations, system 100 i) processes Based on the type of machine learning task, ii) the neural architecture of the convolutional neural network, iii) the computing environment, iv) the performance objectives, or v) a combination of these, define the number of group convolutions (e.g., partitions). Exemplary computing environments may include cloud-based computing environments or mobile device computing environments. Performance objectives may include speed, latency, hardware utilization, model accuracy, parameter size, or a combination of these.

[0034] Group convolution can be described as a generalized form of convolution. In some implementations, system 100 initializes a control parameter g by assigning a specific value to the control parameter. The initialized or assigned value of the control parameter g can be used to control the partitioning of the group convolution. For example, if system 100 decides that a convolution operation using data across the entire channel dimension (e.g., a full convolution) is required, system 100 sets the value of the control parameter g=1 and triggers and / or performs a full convolution using the relevant data in matrix structure 202.

[0035] Relatedly, system 100 can determine the grouping of depth-separable convolutions required for a given step in a larger neural network computation. For example, if system 100 determines that two or more depth-separable convolutions are needed using some of the data in the channel dimension, system 100 sets the control parameter to the desired value (e.g., g=4) and triggers and / or executes two or more (e.g., four) depth-separable convolutions using the relevant data portion in the matrix structure 202. In some implementations, the computation of two or more group convolutions is performed sequentially, concurrently, or in a combination thereof. For example, some (or all) of the respective sets of computations for each of the two or more depth-separable convolutions can be executed sequentially or in parallel.

[0036] As described above, the group / grouped convolution techniques described in this document provide at least finer control over hardware resource utilization metrics and computational efficiency for exemplary ML accelerators. In some implementations, these group convolution techniques provide a multipurpose block or control knob used to influence and control certain attributes or performance metrics of exemplary machine learning models. For example, selecting a value for a control parameter g between 1 and the number of channels (z) provides a continuum between two exemplary constraints: full convolution and convolution separable into depth units. This is discussed in more detail below.

[0037] Figure 3 shows exemplary attributes of a machine learning model. Generally, the attributes correspond to the various convolution operations performed using the convolutional neural network 102 described above. For example, attribute 302 indicates the number of parameters and multiplicative cells (MAC) used to perform a full convolution operation, attribute 304 indicates the number of parameters and multiplicative cells used to perform a depth-unit convolution operation, and attribute 306 indicates the number of parameters and multiplicative cells used to perform a group convolution operation.

[0038] The control parameter g and the configuration of the group convolution can be determined and / or adjusted to control the number of parameters (e.g., trainable parameters) used for a given task, as well as the number of multiplication-accumulation cells used to perform the task's operations. Each of these exemplary attributes of a machine learning model 302, 304, 306 has, or may have, an effect corresponding to various performance metrics of the model. For example, increasing or decreasing the number of trainable parameters and / or multiplication-accumulation cells (or operations) has an effect corresponding to the accuracy, speed, and / or latency of the machine learning model. In another example, the use of depth-unit convolution is lighter than full convolution. While it can be a low-cost (i.e., resource-intensive) option, performing depth-based convolutions on ML accelerator integrated circuits often degrades the utilization of the circuit's hardware resources.

[0039] For example, when performing a depth-unit (or depth-separable) convolution, a standard hardware array of circuit 114 containing tens or hundreds of hardware multiplication-accumulation cells may have a utilization of 3% of those hardware cells in a given computation cycle, while latency is minimal or minimal. Thus, while depth-unit convolution can be fast, it is also inefficient due to the low hardware utilization. Conversely, when performing a full convolution, the hardware array of circuit 114 may have a significantly higher utilization (e.g., 73%), resulting in a large portion of the array's multiplication-accumulation cells being used in a given computation cycle. Compared to depth-unit convolution, this higher utilization when performing a full convolution often comes at the cost of significantly longer computation latency.

[0040] As described above, the group convolution technique described in this document provides finer control over the hardware resource utilization metrics and computational efficiency of an exemplary ML hardware accelerator. By selecting a value for the control parameter g between 1 and the number of channels (z), a continuum is realized between two exemplary constraints: full convolution (308) and convolution separable into depth units (310). System 100 can determine the partitioning of the group convolution by referring to the depth level of the channel dimension, as shown in the example in Figure 2. The control parameter g is used to define the number of group convolutions (e.g., partitions).

[0041] The exemplary graph 312 in Figure 3 shows an exemplary number of parameters 320 and number of MACs 322 for the selection of different values ​​of g (324) between 2 and the number of channels (z) along the continuum between full convolution (308) and depth unit convolution (310). In this example, the zin dimension is 256. Graph 312 shows an example of the decrease in the number of trainable parameters and the number of multiplicative cumulative cells (or operations) for a corresponding increase in the value (g) of the group convolution.

[0042] As described above, circuit 114 may include memory with locations corresponding to tensor elements that can be traversed or accessed using the circuit's control logic to compute the outputs of layers such as group convolutional layers. The elements retrieved from memory (e.g., inputs or activations) must be useful for computing the multiple outputs of the layer. The number of weights (i.e., parameters) also varies depending on the size of the grouping. In some implementations, parameter transfer from memory can become a bottleneck, increasing computation latency. Bottlenecks related to parameter transfer time can be identified from an exemplary set of search data or simulations when determining a preferred neural network architecture. Then, using the disclosed group convolution concept and group convolution-based neural blocks, we can define architectures that reduce the number of parameters and improve or accelerate computation time for machine learning tasks.

[0043] Figure 4 is a block diagram showing examples of process blocks 410, 420, and 430. Each process block 410, 420, and 430 contains one or more layer blocks. In general, each process block 410, 420, and 430 can be represented by different layer blocks of a convolutional neural network. In the example in Figure 4, each of process blocks 410, 420, and 430 may be a subset of operations performed for a given convolution operation. The convolution operation is performed using a convolutional neural network 102, which can be implemented on the exemplary hardware integrated circuit 114 described above.

[0044] A neural network block can describe a component of a neural network, either a single layer or multiple layers. A common block widely used in exemplary computer vision models, such as mobile vision models, is the inverse bottleneck (IBN) layer block 402 ("IBN layer 402"). Generally, an IBN block can be a macroblock of a larger neural architecture, combining multiple convolutional layers in a certain way. Multiple types of layers (or blocks), including IBN layers, are used as building blocks to form exemplary classification or object detection networks.

[0045] The IBN layer 402 can include a point-by-point convolution (404), a K×K depth-by-point convolution (405), and a final point-by-point convolution (406). The point-by-point convolution extends the channel dimension, an example of which is shown in Figure 4 as "1×1 convolution (extended)". The K×K depth-by-point convolution kernel is applied to the depth to which the channel dimension is extended, following the point-by-point convolution. The final point-by-point convolution (406) projects the extended channel dimension to a smaller value. An example of this final point-by-point convolution is shown in Figure 4 as "1×1 convolution (projection)".

[0046] The use of K×K depth-unit convolutions, such as in IBN layer block 402, is quite common. This is because, after expansion, computing a full convolution across large or expanding channel dimensions is very costly in terms of processing and computational resources. In some implementations, point-unit convolutions (404) and K×K depth-unit convolutions (405) are replaced by a K×K full convolution (fused expansion) process block representing a fused IBN layer 407. Generally, a fused IBN layer 407 integrates expansion and depth-unit convolution operations into a single full convolution neural block.

[0047] Full convolution can involve a large number of parameters / weights and may require a significant proportion of the integrated circuit's hardware computational resources. As mentioned above, examples of such resources include the multiplication and accumulation cells of the hardware computation array (e.g., systolic array) of circuit 114, the vector device of integrated circuit 114, or both. In contrast, the disclosed group convolution techniques, implemented using disclosed neural block alternatives such as blocks 414, 416, 422, and 432 described below, offer an improved approach to increasing the number of trainable parameters for a set of input channels (e.g., large input channels), thereby improving the accuracy of the model, but at a lower computational cost compared to non-group convolution alternatives.

[0048] Referring here to process block 410, a grouped IBN gradual projection (or gradual extension) block is shown, where the K×K depth unit convolution (405) described above is replaced with a K×K group convolution (414) or (416). Process block 410 may have a first example that implements a K×K group convolution (414) to perform a gradual projection of the channel dimension, or a second example that implements a K×K group convolution (416) to perform a gradual extension of the channel dimension.

[0049] In the first example of process block 410, system 100 can generate an extended feature map from an input feature map (e.g., input 438) by applying a 1x1 convolution (extension) (404) to the input feature map. The input feature map may be an hxw feature map with a c1 channel. This extended feature map may be an hxw feature map with a c2 channel, where c2 is greater than c1. In some implementations, the 1x1 convolution has more output filters than input filters. A KxK group convolution (414) is applied to the extended feature map and performs a gradual projection of the channel dimensions. For example, a convolutional neural network 102 can perform a gradual projection onto the extended feature map using a group convolution implemented in the group convolution layer of the convolutional neural network 102. Progressive projections can be performed. Grouped IBN progressive projections can offer the flexibility to trade off projection-specific parameters with the main K×K convolution operator.

[0050] In this first example of process block 410, the final point-by-point convolution (406) projects the expanded channel dimension to a smaller value. Thus, the K×K kernel associated with the group convolution can perform an initial reduction of the channel size before the 1×1 projection (406) reduces the channel size to its final value. Each add block 418 is an optional residual (or skip) connection that can be used to add the input 438 supplied to a given process block (e.g., 410) with the exemplary convolution output 436. The exemplary sum 440 is passed as the output of the operation performed in the corresponding process block.

[0051] In a second example of process block 410, system 100 can generate an initial extended feature map from an input feature map (e.g., input 438) by applying a 1x1 convolution (extension) (404) to the input feature map. This initial extended feature map may be an hxw feature map with a c2 channel, where c2 is greater than c1. System 100 generates an extended feature map from the initial extended feature map by applying a KxK group convolution (416) to the initial extended feature map. For example, a convolutional neural network 102 can generate an extended feature map from the initial extended feature map using a group convolution implemented in the group convolution layer of the convolutional neural network 102. The extended feature map may be an hxw feature map with a c3 channel, where c3 is greater than c2. This grouped IBN gradual extension operation can provide flexibility in trading off extension-specific parameters with the main KxK convolution operator. In grouped IBN incremental expansion, some of the expansion layers may be kept unfused, allowing for channel-level convolutions between groups before the main K×K convolution. The final point-level convolution (406) in process block 410 projects the expanded channel dimension to a smaller value.

[0052] Referring now to process block 420, this process block is a fused grouping IBN block in which the aforementioned 1×1 convolution (extended) (404) and K×K depth unit convolution (405) are replaced by a K×K group convolution (422). This K×K group convolution (422) includes the designation "fused extended" because it replaces at least the point unit (404) + depth unit (405) pairs and extends the channel dimension by fusing the modes of those operations via the K×K group convolution (422). Thus, in process block 420, system 100 can generate an extended feature map from an exemplary input feature map (e.g., input 438) by applying the K×K group convolution (422) to the input feature map. The exemplary input feature map may be an h×w feature map with a c1 channel. The extended feature map may be an h×w feature map with a c2 channel, where c2 is greater than c1. The final point-by-point convolution (406) of process block 420 projects the expanded channel dimension to a smaller value. As previously mentioned, the corresponding sum 440 is passed as the output of a specific operation performed in process block 420.

[0053] In some implementations, the fused group convolution block 422 provides an alternative to the fused IBN layer 407, enabling more efficient processing along the channel dimension. For example, these efficiencies can be realized in later stages of the computer vision model. In some cases, these later stages address cases where the data resolution associated with the convolution along the channel dimension is very large. The speed improvements obtained by fused group convolution can be particularly optimized when the process block 420, which includes the group convolution operation, is performed using a specific type of dedicated integrated circuit. For example, a dedicated integrated circuit blows the layer input from memory to one or more computation cells of the circuit, as described below with reference to Figure 6. It may also be a neural network processor that includes a broadcast input bus for docasting.

[0054] The fused group convolution block 422 may require a slightly higher number of parameters compared to the grouped IBN layer 414. In the continuum between the two constraints of complete convolution and convolution separable at depth units, the fused group IBN 422 lies above the continuum. For example, the fused grouped IBN layer 422 may be closer to a complete convolution along the continuum from a depth unit convolution to the complete continuum.

[0055] Referring to process block 430, this process block is a grouped IBN block in which the aforementioned K×K depth unit convolution (405) is replaced by a K×K group convolution (432). As described above, system 100 applies a 1×1 convolution (404) to input 438 to generate an augmented feature map. The K×K group convolution (432) is applied in the group convolution layer of the convolutional neural network 102. The K×K group convolution (432) can have the same total number of input filters and output filters. As with other process blocks, the final point unit convolution (406) of process block 430 projects the augmented channel dimension to a smaller value, and the corresponding sum 440 is passed as the output of a particular operation performed in process block 430.

[0056] The convolution operations performed in process block 430 can have smaller scaling factors compared to the baseline IBN layer. These smaller scaling factors can reduce the number of parameters. To recover the number of parameters, the convolution operations in process block 430 (and other process blocks) can use K×K kernel group convolutions that leverage inter-channel information. The K×K group convolution (432) can be interleaved with other block types that include convolutions along the input channel dimensions. This interleaving pattern can mitigate the shortage of input channel convolutions between groups.

[0057] In general, the architectures of process blocks 410 and 430 replace K×K depth-unit convolutions with K×K group convolutions. At least one advantage of replacing K×K depth-unit convolutions with K×K group convolutions is that K×K group convolutions have lower latency and generate more trainable parameters compared to full convolutions. The additional trainable parameters resulting from the use of K×K group convolutions contribute to improved model accuracy. This improvement in accuracy can be achieved with only a slight or minimal increase in latency compared to depth-unit convolutions.

[0058] Replacing depth-unit convolution with group convolution may be specific to convolution operations in certain types of hardware accelerators, such as tensor processing units (TPUs) configured for mobile devices or edge computing applications. In some implementations, K×K group convolution can be configured to provide a more efficient hardware mapping with respect to the hardware layout of the integrated circuit 114 compared to K×K depth-unit convolution. For example, instead of a one-to-one relationship between input and output channels, group convolution can leverage the concept of blocks to perform convolution along the input channels within a group. This provides the algorithmic advantage of being able to use more information along the input channels, improving the expressive power of one or more layers in a computer vision network.

[0059] As computations for a machine learning task progress to deeper layers of a CNN, the channel dimension can increase. To achieve improvements in certain performance aspects such as output accuracy and computation / processing speed... Conventional approaches have considered using fused IBN layer blocks such as the fused IBN layer 407 mentioned above. However, the large dimensions of each input channel (zin) make the use of fused IBN layers impractical due to the cost of performing a full convolution and the resulting decrease in computational speed.

[0060] Compared to conventional approaches, the group convolutions of process blocks 410, 420, and 430 each provide alternatives to neural blocks that can improve model performance while minimizing certain processing penalties. For example, the fused grouped IBN block 422 can be used to achieve performance improvements without the latency and augmentation / large dataset processing penalties associated with conventional IBN layers or fused IBN layers. In general, each of the group convolution blocks 414, 416, 422, and 432 is a neural network block that can contain one or more group convolutional layers. Furthermore, each group convolution block 414, 416, 422, and 432 can be interleaved with other layers or block types that implement convolution along the input channel dimension. An example of interleaved neural blocks is shown in Figure 5.

[0061] Interleaved patterns can mitigate the lack of input channel convolution between groups. For example, group convolution uses interchannel information, but such information is limited to groups only, and when using groups, shuffle operations are usually required to mix the information along the channel dimension. Interleaved patterns also avoid the use of these additional shuffle operators (e.g., ShuffleNet). Similar to blocks 410 and 430, fused group convolution operations via, for example, block 422 can generate more trainable parameters compared to baseline IBNs, and for certain types of tensor shapes, can improve processing speed compared to baseline IBN and fused IBN layers (e.g., faster execution speed).

[0062] In some implementations, depth-based convolution restricts input and output channels to the same size, while group convolution can enable different sizes. For example, a K×K group convolution (414) kernel can perform an initial reduction in channel size before the 1×1 projection reduces the channel size to its final value. One assumption here is that once group convolution reduces the channels to the final channel dimension, thereby eliminating the 1×1 projection, the performance may be below optimal (e.g., degraded) due to the small channel depth (zo) per group. However, this can be mitigated if group convolution is natively supported by an integrated circuit configuration that allows for the implementation of gradual expansion. For example, the circuit configuration may include an input bus that can pass inputs to separate MACs of the integrated circuit. This is discussed in more detail below with reference to Figures 6-9.

[0063] System 100 is operable to select from several different types of group convolution blocks. For example, in addition to the group convolution blocks 414, 416, 422, and 432 described above, System 100 can also select a fused projection grouping convolution block that implements K×K group convolution. A fused projection grouping convolution fuses point-level projections into a K×K main convolution (instead of fusing point-level extensions). Depending on the tensor shape, a fused projection grouping IBN can provide more trainable parameters while achieving similar processing efficiency compared to a fused IBN. A fused projection grouping IBN retains some of the projection layers without fusing them, allowing for channel-level convolution between groups after the main K×K convolution.

[0064] Figure 5 shows an exemplary architecture of a convolutional neural network 500 for machine learning model 102 that can be used in the exemplary computing system of Figure 1. The Architecture 500 can implement multiple sets of convolution operations to obtain various characteristics of an exemplary input image. In some implementations, System 100 can computationally select and arrange various IBN layer / block options from the grouped and ungrouped IBN options described above with reference to the example in Figure 4. In some implementations, System 100 can computationally select and arrange operations in stacked, connected, or combined configurations (i.e., arrange and combine them together) to form an exemplary Architecture 500 that can be used to implement large-scale computer vision networks / models.

[0065] In the example in Figure 5, architecture 500 includes a sequence of layer blocks, each of which in the first subset of layer blocks in the sequence is configured to perform operations for processing an input image. More specifically, architecture 500 includes a first subset of layer blocks 502, a second subset of layer blocks 504, and a third subset of layer blocks 506. In some implementations, at least one subset of layer blocks 502, 504, and 506 may contain alternating sequences of two or more different types of neural blocks. For example, a subset of layer block 502 may have an alternating sequence of fused IBN layers and fused group IBN layers.

[0066] A fused IBN layer can represent a first individual neural block 512, such as a fused IBN layer 407 (above), which integrates extension and depth-unit convolution operations into a single fully convolutional neural block, while a fused group IBN layer can represent a second individual neural block 514, such as a fused group IBN 422, which replaces point-unit (404) + depth-unit (405) pairs and allows the extension of the channel dimension by fusing the aspects of those operations via a K × K group convolution (422). As described above, this block can provide an alternative to the fused IBN layer 407, enabling more efficient processing along the channel dimension.

[0067] More specifically, the first neural block 512 may be an ungrouped IBN block, and the second neural block 514 may be a grouped IBN block. Each of the first neural block 512 and the second neural block 514 contains one or more convolutional neural network layers. Thus, the layer block 502 may contain alternating sequences of grouped IBN layers and ungrouped IBN layers. For example, the alternating sequence of layer blocks may include ungrouped convolutional layer blocks and interleaved grouped convolutional layer blocks.

[0068] Figure 6 shows an exemplary hardware computation tile 600 ("computation tile 600") used to perform computations on a convolutional neural network. Multiple computation tiles 600 can be arranged or configured to form a dedicated processor such as a neural network processor, an application-specific integrated circuit, or a hardware accelerator. In some implementations, computation tile 600 is one of several computation tiles included in the hardware integrated circuit 114 described above.

[0069] Each computation tile 600 is configured to independently perform computations required by one or more layers of a multilayer neural network (e.g., neural network computations). For example, a computation tile 600 is configured to execute multiple computation threads based on data and instructions retrieved locally from the computation tile 600's memory (described later). In some cases, data and instructions are received by the computation tile 600 via the communication / data bus 602 of the hardware integrated circuit 114. For example, the data bus 602 is coupled to each computation tile 600 to route data between different computation tiles 600 and compute instructions. Therefore, in the case of a given compute tile 600, data and instructions can be received by compute tile 600 from a source outside the tile. The source could be another compute tile 600, a higher-level controller of the hardware circuit 114, a host device outside the hardware circuit 114, or a combination of these.

[0070] The computation tile 600 receives a set of data 604 that may contain instructions and operands for performing a neural network computation. As described below, the data 604 may contain instructions and operands for performing a group convolution operation. The computation tile 600 uses local control logic (e.g., a controller) to identify the instructions and operands in response to its analysis of the data 604. The control logic generates control signals for processing the operands based on one or more instructions. For example, the control logic uses one or more opcodes of the instructions to generate control signals for each corresponding component of the computation tile 600. The components work together based on the control signals to perform the group convolution operation.

[0071] In the example in Figure 6, the local control logic is represented by at least a tensor control unit 606 ("tensor control 606") and a memory access control unit 608 ("DMA control 608"). Tensor control 606 includes a tensor traversal unit (TTU) 626. Generally, tensor control 606 uses the TTU 626 to manage tensor traversal operations for neural network computations, which will be explained in more detail below. DMA control 608 manages writing / storing operands of a given computation to memory locations in local memory contained within the computation tile 600. DMA control 608 also manages reading / retrieving operands of a given computation from memory locations in local memory. In some implementations, DMA control 608 works in conjunction with the TTU 626 to perform memory access operations. In some other implementations, DMA control 608 includes a dedicated TTU for performing memory access operations independently of its interaction with the TTU 626.

[0072] Each computation tile 600 contains memory for storing inputs to neural network layers and for storing layer weights. The inputs and weights correspond to operands (or data) arriving at the computation tile 600 via the communication bus 602. In the example in Figure 6, the memory includes a first memory 610 for storing inputs to neural network layers and a second memory 612 for storing the weights of the neural network layers. The first memory may be a narrow memory that stores, reads, or otherwise manages data in chunks of, for example, 8 bits, while the second memory may be a wide memory that stores, reads, or otherwise manages data in chunks of, for example, 32 bits. The first and second memories can each store, read, and manage data with more or fewer bits. In some implementations, each of the first memory 610 and the second memory 612 is a sub-part of the larger local memory of the computation tile 600. In some other implementations, the first memory 610 and the second memory 612 are each separate local memory devices for the computation tile 600.

[0073] Each calculation tile 600 contains a calculation device 614 configured to perform arithmetic operations such as addition and multiplication using operands corresponding to the input and weight values ​​passed to the calculation tile 160. Each calculation device 614 can contain multiple arithmetic blocks. In the example in Figure 6, each arithmetic block is identified as "cell #_". Each arithmetic block (or cell) contains a multiplication-accumulation cell 616 and a sum register 618. The multiplication-accumulation cell 616 is configured to perform arithmetic operations (e.g., multiplication) using its input and weights.

[0074] For example, arithmetic operations involve inputs or activations obtained from narrow memory 610, and This involves multiplying by weights obtained from memory 612 to generate one or more sets of cumulative values. Each computation tile 600 includes its own input bus 617, which allows it to broadcast, pass, or otherwise provide input to a separate block or multiplication-cumulative cell 616 of the computing unit 614. In some implementations, the input bus 617 is a broadcast input bus that broadcasts the inputs of the group convolutional layer from narrow memory to one or more multiplication-cumulative cells. A total register 618 is used to store partial sums that can be grouped to form a set of cumulative output values ​​620.

[0075] Each computational tile 600 includes an output bus 622 and an activator 626 coupled to the output bus 622. Optionally, the computational tile 600 may include one or more registers 624 coupled to the output bus 622. In some implementations, each of the one or more registers 624 is an individual shift register used to shift the output values ​​620 of the neural network layer (e.g., cumulative values ​​or partial sums) to the activator 626. The activator 626 is computationally capable of applying nonlinear activation functions to the output values ​​620. Based on the activation functions applied to the output 620, the activator 626 is computationally capable of generating a set of output activations for the layer.

[0076] The activation device 626 is coupled to the first memory 610 and is configured to pass output activations to the narrow memory 610 for storage in memory. Output activations correspond to the layer outputs of neural network layers. For example, a set of output activations could be the output (or part of the output) of a group convolutional layer that applies group convolution to an input feature map to generate an output feature map. Thus, output activations can correspond to an output feature map. In some implementations, the activation device 626 can perform operations to aggregate multiple partial sums or cumulative values ​​into a vector of values.

[0077] Each computation tile 600 may include an optional group convolution control 635 that can be computed to manage and implement operations on the group convolution layer in the computation tile. For example, a computation tile 600 may receive instructions to process an input set via a group convolution layer by applying a group convolution to one or more input groupings along the channel dimension of an input feature map. The individual inputs of one or more input groupings may be stored not only in various locations in memory 610 but also across different computation tiles 600. Each memory location is identified by its respective address. Each memory location (or its respective address) storing each group convolution input may correspond to an element of an input tensor, such as a multidimensional input tensor or input feature map, stored in the first memory 610.

[0078] The group convolution control 635 can obtain or determine the memory address of the corresponding group convolution input, which is broadcast to one or more multiplication-accumulation cells 616. In some implementations, the group convolution control 635 communicates with the DMA control 608 and interacts with it to issue an address for accessing the memory location of the corresponding group convolution input. In some other implementations, the group convolution control 635 communicates directly with the first memory 610 to access the memory address of the corresponding group convolution input. The group convolution control 635 can perform a similar operation to access the weights of the parameter tensor stored in the second memory 612 and pass the weights to the corresponding multiplication-accumulation cell or load them into the corresponding multiplication-accumulation cell. The group convolution control 635 is described further below with reference to Figure 8.

[0079] Each compute tile 600 is configured to execute one or more compute threads. In some implementations, the hardware circuit 114 is part of the compute tile 600 (or All of these are used to run multiple computation threads in parallel. A computation thread may run over multiple clock cycles and is used to process inputs to a neural network layer and generate the output of the neural network layer. For example, by assigning each subset of computation threads to one or more computation tiles 600, a loop nest of group convolutional layers can be implemented, applying group convolution to an exemplary input feature map. This is explained in more detail below. Figure 6 includes a reference map 630 showing the attributes of each of the various components within the computation tile 600. The reference map 630 is shown for clarity and is not included in the computation tile 600. Attributes include whether a particular component is a device, memory device, operator, control device, or data path.

[0080] Figure 7A is a block diagram illustrating an exemplary mapping of partitions to a subset of multiplication and accumulation cells 616. Figure 7B is a block diagram illustrating an exemplary input bus 617 that provides respective inputs to the multiplication and accumulation cells 616 of the hardware computation tile 600.

[0081] Referring first to Figure 7A, as described above, data and instructions can be received by the compute tile 600 from sources outside the tile. The source may be another compute tile 600, a higher-level controller of the hardware circuit 114, a host device outside the hardware circuit 114, or a combination of these. Based on the type of group convolution operation to be performed, the system 100 can select from a variety of predefined values ​​for the control parameter g, which represents the number of group convolutions (e.g., partitions). For example, the system 100 can select a specific value of g for different group convolution neural blocks of a given neural network architecture. In some implementations, the value of g is predefined on an external host for a given operation and passed to the controller of the hardware circuit 114.

[0082] In some implementations, a higher-level controller identifies one or more partitions along the channel dimension (e.g., Zin) of the input feature map based on a control parameter g. System 100 can then form one or more groupings along the channel dimension based on one or more partitions. In the example in Figure 7, each grouping of the input channels is formed along Zin of the exemplary input tensor or input feature map. Each grouping can be mapped to the corresponding multiplication-accumulation cells 616-1, 616-2, 616-3, and 616-4, as described below. Furthermore, each grouping of the input channels includes its own size. More specifically, each grouping includes a number of input channels corresponding to the respective size of the grouping. For example, as shown in Figure 7A, the size parameter S of a grouping or partition can be defined by Zin / g, where Zin is the number of input channels along the channel dimension of the input tensor and g is the number of groups defined by the control parameter described above.

[0083] System 100 is capable of determining the mapping 700 of groupings to the multiplication accumulating cell 616 in the computing device 614. For example, the mapping may be determined locally at the computing tile 600 or using a higher-level controller on the integrated circuit 114. In some implementations, the host device determines the mapping, generates a mapping instruction, passes the mapping instruction to a higher-level controller, and the higher-level controller then passes the instruction to the computing tile 600. For example, the integrated circuit 114 may include a host interface block for receiving data or instructions passed from an external host device to a higher-level controller.

[0084] In some implementations, system 100 (e.g., host or integrated circuit controller) determines the mapping based on the number of channels in each partition. For example, Either a storage device or a higher-level controller can access information describing the hardware configuration of the integrated circuit 114, including the configuration of the computing devices 614 within each computing tile 600. Based on this hardware configuration, system 100 can determine the size of each grouping by referring to the number or layout of multiplication accumulating cells in computing device 614. For example, system 100 can determine the optimal mapping of groupings to their respective inputs in multiplication accumulating cells 616 to maximize the overall utilization of computing device 614. This will be explained in more detail below.

[0085] Next, referring to Figure 7B, an exemplary architecture is shown in which an input bus 617 coupled to the narrow memory 610 broadcasts inputs / activations to one or more multiplication accumulator cells 616. Inputs can be shifted or sent to the input bus 617 one at a time so as to be received by the corresponding multiplication accumulator cells 616.

[0086] In some implementations, the input bus 617 is a broadcast input bus that broadcasts the group convolutional layer inputs obtained from the narrow memory 610 to one or more multiplication-accumulation cells 616. For example, the input bus 617 can pass (or broadcast) its respective inputs to individual multiplication-accumulation cells 616-1, 616-2, 616-3, and 616-n of the integrated circuit 114. Thus, the input bus 617 includes a broadcast function that allows the integrated circuit 114 to broadcast the multiple inputs of each grouping along the Zin dimension of the input feature map to the corresponding multiplication-accumulation cells 616, based on the determined mapping described above.

[0087] In some implementations, the same input is shared among several (or all) of the multiplication-accumulation cells 616 within a subset of cells 616. The width of the input bus 617 must be wide enough to supply inputs broadcast to a number of multiplication-accumulation cells 616 corresponding to a given subset of the computing device 614. For example, with respect to the structure of the input bus 617, if the number of multiplication-accumulation cells 616 in the computing device 614 is 4 and the data resolution / width of the input (or activation) is 8 bits, the input bus 617 can be configured to provide up to 4 input activations per cycle. In this example, each multiplication-accumulation cell 616 can receive one of the 4 activations broadcast.

[0088] System 100 can broadcast each first input ("0") of the first grouping (along Zin) of the input feature map to each multiplicative accumulator cell 616-1, 616-2, 616-3, 616-n within the subset of multiplicative accumulator cells 616-616 via the input bus 617. Similarly, System 100 can broadcast each second input ("1") of the second grouping (along Zin) of the input feature map to each multiplicative accumulator cell 616-1, 616-2, 616-3, 616-n within the subset of multiplicative accumulator cells 616-616 via the input bus 617. The broadcasted first and second inputs are reused during the computation of the group convolutional layer. For example, each input 702 ("0"), 704 ("1"), 706 ("2"), 708 ("3") can correspond to different groupings along the channel dimension of the activation tensor.

[0089] In some implementations, each input 702, 704, 706, and 708 can be broadcast and reused across each multiplication accumulator cell to parallelize the computation of the group convolution layer. For example, to perform part of the group convolution, the reused inputs are retrieved from memory locations in wide memory 612 and multiplied by different individual weight values, which are then routed to the respective weight registers of the multiplication accumulator cell 616. This reuse attribute is described in more detail below with reference to Figure 8. By parallelizing the computation of each Zin grouping in this way, circuit 114 can maximize the utilization of the computing device 614 and the corresponding multiplication accumulator cell 616 within the device. More specifically, at least the multiplication accumulator cells of circuit 114 A circuit architecture for performing group convolution that enables 616-way input broadcasting can achieve utilization and efficiency levels that surpass conventional circuit architectures used to perform group convolution.

[0090] Furthermore, at least one advantage of group convolution-based neural blocks is the ability to change the computational intensity. For example, the computational intensity can be adjusted to control the number of operations performed in the multiplication accumulation cell 616 and the overall cell utilization per weight retrieved. This allows system 100 to optimize parameter bandwidth. In some cases, memory bandwidth can be limited in applications for inference computations at the edge. Group convolution can be used to maximize overall computation time and minimize (or avoid) the need for extra memory operations to retrieve new weights from memory.

[0091] Figure 8 is an exemplary block diagram 800 illustrating certain attributes of full convolution, depth-based convolution, and group convolution. More specifically, block diagram 800 shows the reuse attributes of the inputs processed during the full convolution operation (802), depth-based convolution operation (804), and group convolution operation (806). In the example in Figure 8, reuse is illustrated by referring to blocks 802 (full convolution) and 806 (group convolution). For example, the first block 802 shows that each input 812 is reused to compute each output channel 813 of the full convolution, while the second block 804 shows that each input 814 is used only once to compute the corresponding output channel 815 of the depth-based convolution.

[0092] The third block, 806, demonstrates that inputs can have a reuse factor when computing output channels 817, 818 corresponding to a given group convolution. For example, in block 806, each input 816 has a specific reuse factor (e.g., 2) when computing the corresponding output channels 817, 818. The reuse factor of an input to a group convolution layer corresponds to the size of the groupings processed by that layer. In some cases, each element from an input channel is reused to compute an output channel belonging to that group. Taking this into account, the reuse factor is determined based on the size of the group.

[0093] In some implementations, the first opcode in the instruction received by the computation tile 600 specifies a value for a control parameter g that indicates the partitioning and subsequent grouping of the input tensor group convolution input, and the second opcode in the instruction specifies a value for a size parameter that indicates the input reuse coefficient in the grouping. Each computation tile 600 may also use a local group convolution control 635 to determine the size parameter based on the hardware configuration of the computation tile 600, the group convolution performed on the computation tile 600, or both.

[0094] The circuit 114 may have 32 or 64 multiplication and accumulation cells within the computing device 614. The group convolution control 635 can identify one or more opcodes in an instruction that specifies a group convolution operation on the computing tile 600. For example, the group convolution control 635 may determine that the group convolution is a K × K group convolution (416) and perform a gradual expansion of the channel dimension. The group convolution control 635 may determine that this particular type of convolution operation is applied to one or more group convolution layers of the grouped IBN gradual expansion neural block 416.

[0095] In some implementations, system 100 can select a predefined value of a control parameter g specific to a particular type of convolution operation. For example, given group convolution neural blocks 412, 416, 422, or 432, system 100 can select from a predetermined set of control values ​​g for the different group convolution operations associated with each neural block. Yes, it is possible. The grouping in a group convolution operation is defined by a control value. In some implementations, a group convolution control 635 for a given computation tile 600 determines the local mapping 700 of the grouping to the multiplication accumulating cell 616 of the tile. For each grouping, the group convolution control 635 can identify a group convolution layer of neural blocks 416 to process the group convolution input 816 of the operation and set a size parameter S according to the grouping and operation.

[0096] As mentioned above, each grouping contains the number of input channels corresponding to the size of each grouping, so the size parameter S of a grouping can be defined by Zin / g. Each grouping represents a group convolution and is associated with the corresponding channels of the output groups 220-n. Each grouping can contain the respective inputs derived from the input feature map. In the example in Figure 8, each input 816 may be from a different grouping along the channel dimension of the input tensor. In some implementations, the group convolution control 635 analyzes one or more opcodes of the instruction and, based on the opcodes, decides that the computation tile 600 should apply a K×K group convolution (416) to perform a incremental expansion of the channel dimension, which involves increasing the number of channel dimensions.

[0097] The group convolution control 635 can determine the size parameter S for various aspects of the group convolution and adjust the local mapping of the grouping as the K×K group convolution (416) progresses. In some implementations, this incremental expansion operation is statically represented as part of the neural network. For example, to expand the entire output channel to e, the K×K group convolution (416) can expand the output channel by the coefficient g_e. As described above, a 1×1 point-by-point convolution can be performed after this expansion, which has an expansion of e / g_e, so the overall expansion is g_e*e / g_e=e.

[0098] Next, we will describe an example operation involving group convolution calculations. Referring again to the example of 64 multiplication-accumulation cells, the group convolution control 635 can obtain 64 different weight values ​​for the group convolution layer from the wide memory 612. For example, the group convolution control 635 can obtain 64 different weight values ​​based on the hardware configuration (e.g., the number of cells), the type of group convolution layer, or both. The group convolution control 635 can also obtain the corresponding input 816 from memory 610. The input to be obtained must have reuse across the entire multiplication-accumulation cell 616. In some implementations, the input to be obtained has reuse across the entire 64 multiplication-accumulation cells. In some other implementations, the input to be obtained has reuse across a subset of the 64 multiplication-accumulation cells. In general, all inputs within a group have some degree of reuse for outputs within the same group.

[0099] In this example, input 816 can be selected from an input feature map with an input depth of 64, such that the input depth corresponds to the number of multiplicative cumulative cells. The computation tile 600 can compute 1000 outputs using 64 cells. Since the group convolution control 635 can set the group size to 64, for each cycle that acquires and broadcasts one input value 816, the computation tile 600 can compute 64 of the 1000 outputs using that input 816. Thus, if the group size is sufficiently large, a given computation tile 600 can achieve 100% utilization of the input bus. This is because all 64 cells are used for each cycle that acquires one input value.

[0100] Therefore, the computation tile 600 defines the group size based on the number of multiplicative cells, and depending on certain characteristics of the group convolution, it can achieve full utilization of the multiplicative cells without incurring the processing penalty of a full convolution. Example with a single input channel. If the group size is 2, this means that computation tile 600 convolves two channel elements (for example, two inputs 816). Therefore, computation tile 600 convolves that number of channel elements based on the group size. In the case of a full convolution, the group size is equal to the size of the entire input channel.

[0101] If there are 1000 input channels, in a full convolution, system 100 convolves all 1000 input channels to compute one output channel. Here, the output is a channel of values ​​or activations. In the case of depth-based convolution, system 100 computes only one input channel to compute one output channel. In this example, when the group size is 1, this is depth-based convolution. When the group size is 2, two input channels must be convolved to compute one output channel. When the group size is 4, four input channels must be convolved to compute one output channel.

[0102] Figure 9 is a flowchart of an exemplary process 900 used to process an exemplary image by applying group convolution using a hardware integrated circuit. The hardware integrated circuit is configured to implement a CNN containing multiple neural network layers, the multiple layers of which include group convolutional layers. The exemplary image may be the image 102 described above, or it may be various other types of digital images and associated graphic data. In some implementations, process 900 is part of a technique used to speed up neural network computations, which can also improve the accuracy of the image processing output compared to other data processing techniques.

[0103] Process 900 can be implemented or executed using the system 100 described above. Therefore, the description of Process 900 may refer to the computing resources of the system 100 described above. Steps or actions of Process 900 can be enabled by programmed firmware or software instructions executable by one or more processors of the devices and resources described in this document. In some implementations, steps of Process 900 correspond to methods for using a hardware integrated circuit to perform computations to generate the output of a convolutional neural network layer, such as a group convolutional layer. The integrated circuit can be a dedicated neural network processor or a hardware machine learning accelerator configured to implement a CNN.

[0104] Referring again to process 900, system 100 identifies control parameters associated with the input feature map (902). For example, control parameters defining two or more partitions along the channel dimension of the input feature map are identified. System 100 determines the mapping of the two or more partitions (904). More specifically, system 100 determines the mapping of the partitions to multiplication accumulator cells in the computing device of the hardware integrated circuit.

[0105] For a group convolutional layer, system 100 applies a group convolution to the input feature map using a hardware integrated circuit (906). For each of two or more partitions, applying a group convolution to the group convolutional layer includes providing weights for the group convolutional layer to a subset of the multiplicative accumulator cells (908). For example, system 100 provides weights to a subset of the multiplicative accumulator cells based on a determined mapping. The weights are provided from an exemplary wide memory of the computation tile 600.

[0106] System 100 provides inputs to an input feature map to a subset of multiplication and accumulation cells (910). For example, each input of the input feature map is received via the input bus of the integrated circuit. This is provided to each multiplication accumulator cell in the set. More specifically, each hardware computation tile 600 includes its own input bus, which is used to broadcast one or more inputs to a given multiplication accumulator cell.

[0107] System 100 calculates the product using the corresponding weights of each input and the group convolutional layer (912). For example, the product is calculated using a multiplication circuit of multiplication accumulator cells, by multiplying each input and its corresponding weight in each multiplication accumulator cell within the subset.

[0108] System 100 generates an output feature map of the group convolutional layer (914). For example, the output feature map of the group convolutional layer is generated based on the accumulation of multiple respective products calculated in each multiplication accumulation cell 616 within a subset of multiplication accumulation cells. The computation process performed within the computation tile 600 of the group convolutional layer involves multiplying the data values ​​stored in each element of the input tensor (e.g., inputs or activations) with the data values ​​stored in each element of the parameter tensor (e.g., weights). For example, the computation involves multiplying the input or activation values ​​and weight values ​​in one or more cycles to generate multiple products (e.g., partial sums), and then accumulating those products over many cycles. In some implementations, generating the output feature map involves generating the output feature map based on the size of each grouping (or partition) of the input channel.

[0109] Embodiments of subject matter and functional operations described herein can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, or one or more combinations thereof, including structures and structural equivalents disclosed herein. Embodiments of subject matter described herein can be implemented as one or more modules of computer program instructions encoded on a tangible, non-temporary program carrier for execution by or control of operations of a data processing device.

[0110] Alternatively or additionally, program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing device. Computer storage media may be machine-readable storage devices, machine-readable storage boards, random or serial access memory devices, or a combination of one or more of these.

[0111] The term "computing system" encompasses all kinds of devices, machines, and equipment for processing data, such as programmable processors, computers, or multiple processors or computers. Devices may include specialized logic circuits such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, devices may also include code that creates the execution environment for the computer program in question, such as processor firmware, protocol stacks, database management systems, operating systems, or any combination of these.

[0112] Computer programs (which may also be called or described as programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages. It can be deployed in any form, either as an Aron program or as a module, component, subroutine, or other device suitable for use in a computing environment.

[0113] Computer programs may, but do not need to, correspond to files in a file system. A program can be stored in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., a file storing one or more modules, subprograms, or parts of code). Computer programs can be deployed to run on a single computer, or on multiple computers located in one site or distributed across multiple sites interconnected by a communication network.

[0114] The processes and logic flows described herein can be executed by one or more programmable computers running one or more computer programs that perform functions by performing calculations on input data and generating outputs. The processes and logic flows can also be executed by dedicated logic circuits such as FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), or GPGPUs (General-Purpose Graphics Processing Units), and the devices can also be implemented as dedicated logic circuits.

[0115] A computer suitable for running computer programs may be based on, for example, a general-purpose microprocessor, a dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Some computer components include a central processing unit for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operationally coupled to receive data from, transfer data to, or both of those storage devices. However, a computer does not necessarily require such devices. Furthermore, a computer can be incorporated into another device, to name just a few examples, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device such as a Universal Serial Bus (USB) flash drive.

[0116] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, CD-ROMs, and DVD-ROMs. Processors and memory may be complemented by or integrated into dedicated logic circuits.

[0117] To enable user interaction, embodiments of the subject matter described herein can be implemented on a computer equipped with a display device for displaying information to the user (e.g., an LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) that allows the user to provide input to the computer. Other types of devices can also be used to provide user interaction. For example, the feedback provided to the user may be visual feedback, auditory feedback, or It can provide any form of sensory feedback, including haptic feedback, and user input can be received in any form, such as acoustic, voice, or haptic input. Furthermore, the computer can interact with the user by sending and receiving documents to and from the user's device. For example, in response to a request received from a web browser, it can send a web page to the web browser on the user's client device.

[0118] Embodiments of the subject matter described herein can be implemented in a computing system including, for example, a backend component such as a data server, or a computing system including a middleware component such as an application server, or a computing system including a frontend component such as a client computer with a graphical user interface or a web browser, through which a user can interact with the implementation of the subject matter described herein, or it can be implemented in any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs") such as the Internet.

[0119] A computing system can include clients and servers. Generally, clients and servers are geographically separated and typically interact via a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with one another.

[0120] While this specification includes many specific implementation details, these should not be interpreted as limiting the scope of the invention or the scope of what can be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable subcombination in multiple embodiments. Furthermore, even if features are described above as functioning in a combination and initially claimed as such, one or more features from the claimed combination may be removed from the combination, and the claimed combination may be directed towards a subcombination or a variation of a subcombination.

[0121] Similarly, while the diagrams depict operations in a specific order, this should not be understood as requiring that such operations be performed in a specific or sequential order as depicted, or that all depicted operations be performed, in order to obtain the desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the program components and systems described can generally be integrated into a single software product or packaged into multiple software products.

[0122] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order to obtain the desired results. As an example, the process shown in the accompanying diagram does not necessarily require the specific order or sequence shown to obtain the desired results. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A method for processing an input image using a hardware integrated circuit configured to implement a convolutional neural network comprising multiple neural network layers, wherein the multiple neural network layers include a group convolutional layer, and the method Identifying control parameters that define multiple partitions along the Channel dimension of the input feature map, Determining the mapping of the multiple partitions to the multiple multiplication and accumulation cells (MACs) within the computing device of the integrated circuit, The group convolutional layer includes applying group convolution to the input feature map, and for each of the plurality of partitions, Based on the determined mapping, the weights of the group convolutional layer are provided to a subset of the plurality of MACs, The input of the input feature map is provided to each MAC in the subset via the input bus of the integrated circuit, This includes, in each MAC within the subset, calculating the product using the respective inputs and the corresponding weights of the group convolutional layer, The aforementioned method, A method comprising generating an output feature map of the group convolutional layer based on the accumulation of products.

2. Determining the mapping of the multiple partitions to the multiple multiplication accumulation cells is: The method according to claim 1, comprising determining the mapping based on the number of channels in each of the plurality of partitions.

3. The method according to claim 2, wherein each of the plurality of partitions includes a number of input channels corresponding to the respective size of the partition.

4. Generating the aforementioned output feature map is, The method according to claim 3, comprising generating the output feature map based on the respective sizes of each partition.

5. Accessing information describing the hardware configuration of the aforementioned computing device, The method according to claim 3, further comprising determining the respective sizes of each partition based on the hardware configuration of the computing device.

6. The input bus includes a broadcast function, and the method is The method according to claim 1, further comprising broadcasting a plurality of inputs of the input feature map to the computing device of the integrated circuit for each partition via the input bus.

7. The method further includes broadcasting the first input of the first partition of the input feature map to each MAC in the subset via the input bus, The method according to claim 6, wherein the first input that is broadcast is reused during the computation of the group convolutional layer.

8. The first partition of the input feature map corresponds to the first partition of the output feature map, The method according to claim 6, wherein the first input has to be reused for the output of the first partition of the output feature map.

9. Generating the aforementioned output feature map is, Calculating multiple products using the subset of the multiple MACs, The method according to claim 1, comprising generating the cumulative product from the plurality of products.

10. A system for processing an input image, wherein the system is Processor and A hardware integrated circuit configured to implement a convolutional neural network including multiple neural network layers, including a group convolutional layer, Executable by the aforementioned processor, Identifying control parameters that define multiple partitions along the Channel dimension of the input feature map, Determining the mapping of the multiple partitions to the multiple multiplication and accumulation cells (MACs) within the computing device of the integrated circuit, The group convolutional layer includes applying group convolution to the input feature map, and for each of the plurality of partitions, Based on the determined mapping, the weights of the group convolutional layer are provided to a subset of the plurality of MACs, The input of the input feature map is provided to each MAC in the subset via the input bus of the integrated circuit, This includes, in each MAC within the subset, calculating the product using the respective inputs and the corresponding weights of the group convolutional layer, A system comprising: a non-temporary machine-readable memory device that stores instructions for performing an operation, which includes generating an output feature map of the group convolutional layer based on the accumulation of products.

11. Determining the mapping of the multiple partitions to the multiple multiplication accumulation cells is: The system according to claim 10, comprising determining the mapping based on the number of channels in each of the plurality of partitions.

12. The system according to claim 11, wherein each of the plurality of partitions includes a number of input channels corresponding to the respective size of the partition.

13. Generating the aforementioned output feature map is, The system according to claim 12, comprising generating the output feature map based on the respective sizes of each partition.

14. The above operation is, Accessing information describing the hardware configuration of the aforementioned computing device, The system according to claim 12, further comprising determining the respective sizes of each partition based on the hardware configuration of the computing device.

15. The aforementioned input bus includes a broadcast function, and the calculation is performed by The system according to claim 10, further comprising broadcasting a plurality of inputs of the input feature map to the computing device of the integrated circuit for each partition via the input bus.

16. The above operation is, The method further includes broadcasting the first input of the first partition of the input feature map to each MAC in the subset via the input bus, The system according to claim 15, wherein the first input that is broadcast is reused during the computation of the group convolutional layer.

17. The first partition of the input feature map corresponds to the first partition of the output feature map, The system according to claim 15, wherein the first input has reuse for the output of the first partition of the output feature map.

18. Generating the aforementioned output feature map is, Calculating multiple products using the subset of the multiple MACs, The method according to claim 1, comprising generating the cumulative product from the plurality of products.

19. A non-temporary, machine-readable storage device for storing instructions for processing an input image using a hardware integrated circuit configured to implement a convolutional neural network comprising multiple neural network layers, including a group convolutional layer, wherein the instructions are executable by a processor. Identifying control parameters that define multiple partitions along the Channel dimension of the input feature map, Determining the mapping of the multiple partitions to the multiple multiplication and accumulation cells (MACs) within the computing device of the integrated circuit, The group convolutional layer includes applying group convolution to the input feature map, and for each of the plurality of partitions, Based on the mapping determined above, the weights of the group convolutional layer are provided to a subset of the plurality of MACs, The input of the input feature map is provided to each MAC in the subset via the input bus of the integrated circuit, This includes, in each MAC within the subset, calculating the product using the respective inputs and the corresponding weights of the group convolutional layer, A non-temporary, machine-readable storage device that performs an operation including generating an output feature map of the group convolutional layer based on the accumulation of products.

20. The non-temporary machine-readable storage device according to claim 19, wherein each of the plurality of partitions includes a number of input channels corresponding to the respective size of the partition.

Images