Flexible accelerator for sparse tensors in convolutional neural networks

Abstract

A system having a multiplication circuit having a plurality of multipliers is disclosed. Each of the plurality of multipliers is configured to receive a data value and a weight value to generate a product value in a convolution operation of a machine learning application. The system further includes an accumulator configured to receive the product value from each of the plurality of multipliers and a register bank configured to store an output of the convolution operation. The accumulator is further configured to receive a portion of values stored in the register bank and to combine the received portion of values with the product values to generate combined values. The register bank is further configured to replace the portion of values with the combined values.

Description

Background Technology

[0001] The applicant provides the following explanation to aid reader comprehension. The information provided and any references cited are not considered prior art.

[0002] Machine learning algorithms are widely used in fields such as medical imaging, computer vision, advertising, and gaming. These algorithms process massive amounts of input data. As the volume of input data increases, the computational time and resources required to process it also increase. However, the current mechanisms used in machine learning applications have limitations in terms of configuration and the way they process input data. Summary of the Invention

[0003] According to some aspects of this disclosure, a system is disclosed. The system includes a multiplication circuit having a plurality of multipliers. Each of the plurality of multipliers is configured to receive data values ​​and weight values ​​to generate a product value in a convolution operation of a machine learning application. The system also includes an accumulator and a register set, the accumulator being configured to receive the product value from each of the plurality of multipliers, and the register set being configured to store the output of the convolution operation. The accumulator is further configured to receive a portion of a value stored in the register set and combine the received portion of the value with the product values ​​to generate a combined value. The register set is further configured to replace the portion of the value with the combined values.

[0004] According to several other aspects of this disclosure, a method is disclosed. The method includes having a processor in a machine learning application input data values ​​and weight values ​​into each of a plurality of multipliers to generate a plurality of product values ​​in each iteration of a plurality of iterations of a convolution operation. The method further includes having the processor combine each of the plurality of product values ​​with one of a plurality of accumulator values ​​in the plurality of accumulators in each iteration of the plurality of iterations to generate a plurality of combined values. The plurality of accumulator values ​​are received from a register set. The method further includes having the processor replace the plurality of accumulator values ​​with the plurality of combined values ​​in the register set in each iteration of the plurality of iterations.

[0005] According to other aspects of this disclosure, a non-transitory computer-readable medium having computer-readable instructions stored thereon is disclosed. These computer-readable instructions, when executed by a processor associated with a machine learning application, cause the processor to: divide an input feature map into a plurality of sub-feature maps, and input each of the plurality of sub-feature maps into a tensor computation unit among a plurality of tensor computation units to generate an output sub-feature map. Generation of the output sub-feature map for a first sub-feature map of the plurality of sub-feature maps includes: inputting a plurality of data values ​​of the first sub-feature map into a plurality of multipliers in a first tensor computation unit among the plurality of tensor computation units; inputting weight values ​​into the plurality of multipliers to generate a plurality of product values; combining each of the plurality of product values ​​with a product value from a previously calculated product value to obtain a plurality of combined values; shifting the plurality of combined values ​​to obtain the output sub-feature map for the first sub-feature map; and combining the output sub-feature map from each of the plurality of tensor computation units to obtain the output feature map.

[0006] The foregoing description of the invention is merely illustrative and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, other aspects, embodiments, and features will become apparent from the following drawings and detailed description. Attached Figure Description

[0007] Figure 1 This is an exemplary block diagram of a computing system according to some embodiments of the present disclosure.

[0008] Figure 2 It is based on some implementation schemes of this disclosure. Figure 1 An exemplary block diagram of an accelerator for a computing system.

[0009] Figure 3A and Figure 3B This is an example of converting an input feature map into multiple sub-feature maps according to some embodiments of this disclosure.

[0010] Figure 4 It is based on some implementation schemes of this disclosure. Figure 2 An exemplary block diagram of a sparse tensor memory cluster for an accelerator.

[0011] Figure 5 It is based on some implementation schemes of this disclosure. Figure 2 Another exemplary block diagram of the sparse tensor memory cluster of the accelerator.

[0012] Figure 6 It is based on some implementation schemes of this disclosure. Figure 4 and Figure 5An exemplary block diagram of a sparse tensor feature map memory cell of a sparse tensor memory cluster.

[0013] Figure 7 It is based on some implementation schemes of this disclosure. Figure 4 and Figure 5 An exemplary block diagram of a sparse tensor weighted memory for a sparse tensor memory cluster.

[0014] Figure 8 It is based on some implementation schemes of this disclosure. Figure 2 An exemplary block diagram of a sparse tensor computing cluster for an accelerator.

[0015] Figure 9 It is based on some implementation schemes of this disclosure. Figure 2 Another exemplary block diagram of the sparse tensor computing cluster of the accelerator.

[0016] Figure 10 It is based on some implementation schemes of this disclosure. Figure 8 and Figure 9 An exemplary block diagram of a sparse tensor computation unit in a sparse tensor computation cluster.

[0017] Figure 11 This is based on some implementation schemes of this disclosure. Figure 10 An example of a sub-feature map in a sparse tensor computational unit.

[0018] Figure 12 This is an overview of some embodiments of the present disclosure for processing Figure 10 An exemplary flowchart of the operation of sub-feature mapping graphs in a sparse tensor computation unit.

[0019] Figure 13 It is based on some implementation schemes of this disclosure. Figure 8 and Figure 9 Another example of a sparse tensor computation unit in a sparse tensor computation cluster.

[0020] Figure 14 It is based on some implementation schemes of this disclosure. Figure 8 and Figure 9 Another example of a sparse tensor computation unit in a sparse tensor computation cluster.

[0021] Figure 15 It is based on some implementation schemes of this disclosure. Figure 8 and Figure 9 Another example of a sparse tensor computation unit in a sparse tensor computation cluster.

[0022] Figure 16This is an overview of some embodiments of the present disclosure for processing in standard convolution operations or 1×1 fully connected convolution operations. Figures 13 to 15 An exemplary flowchart of the operation of sub-feature mapping graphs in a sparse tensor computation unit.

[0023] Figure 17 This is an overview of some embodiments of the present disclosure for use. Figure 2 An exemplary flowchart of the operation of the accelerator in processing the input feature map.

[0024] Figures 18A to 18D Processing according to some embodiments of this disclosure is shown. Figure 13 An example of a sub-feature map in a sparse tensor computational unit.

[0025] Figure 19A Processing according to some embodiments of this disclosure is shown. Figure 15 An example of a sub-feature map in a sparse tensor computational unit; Figure 19B Processing according to some embodiments of this disclosure is shown. Figure 15 An example of a sub-feature map in a sparse tensor computational unit; Figure 19B (1) is Figure 19B The left side; Figure 19B (2) is Figure 19B The right side; Figure 19C Processing according to some embodiments of this disclosure is shown. Figure 15 An example of a sub-feature map in a sparse tensor computational unit; Figure 19C (1) is Figure 19C The left side; Figure 19C (2) is Figure 19C The right side; Figure 19D Processing according to some embodiments of this disclosure is shown. Figure 15 An example of a sub-feature map in a sparse tensor computational unit; Figure 19D (1) is Figure 19D The left side; Figure 19D (2) is Figure 19D The right side; Figure 19E Processing according to some embodiments of this disclosure is shown. Figure 15 An example of a sub-feature map in a sparse tensor computational unit; Figure 19E (1) is Figure 19E The left side; Figure 19E (2) is Figure 19E The right side of.

[0026] Figures 20A to 20B Examples of merging operations according to some embodiments of this disclosure are shown.

[0027] The above and other features of this disclosure will become apparent from the accompanying drawings, the following description, and the appended claims. It should be understood that these drawings depict only a few embodiments according to this disclosure and are therefore not intended to be limiting of its scope, which will be described more specifically and in more detail through the use of the drawings. Detailed Implementation

[0028] In the following detailed description, reference is made to the accompanying drawings, which form a part of this detailed description. In the drawings, like reference numerals generally identify like parts unless the context otherwise indicates. The exemplary embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments and changes may be utilized without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of this disclosure as generally described herein and illustrated in the figures can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are expressly contemplated and are part of this disclosure.

[0029] As neural networks participate in more complex processing tasks across a wider range of scenarios, they continue to evolve towards larger and deeper architectures. Large neural networks increase computational and memory access as the amount of data increases. Specifically, the number of synapses and neurons in a neural network increases whenever the amount of data to be processed increases. For example, some neural networks may have over 600,000 neurons, while others may have over 1 million neurons. The number of synapses can be much larger; for example, a neural network may have over 60 million synapses. A convolutional neural network (“CNN”) is one type of neural network. Specifically, a CNN is a deep neural network that uses convolution operations to classify inputs (e.g., images). To perform classification, a CNN receives input and assigns weights and biases to that input to distinguish one side of the input from another side. In some implementations, a CNN may include multiple layers for processing between receiving input and sending output. Depending on the size of the input and the number of layers used in the CNN, a large amount of data may need to be processed.

[0030] In accelerator architecture design, handling such large amounts of data presents challenges to both on-chip and off-chip memory bandwidth. Mechanisms can be used to reduce the amount of data to be computed or to reduce computation time. One such mechanism for reducing the amount of data to be computed or computation time can include entropy encoding via quantized weights and reducing the weight size. Other mechanisms operate at the circuit level, such as using shorter bit-width operators or approximation computation. Another mechanism for reducing computation time or data volume relies on sparsity. Sparsity determines the number or percentage of non-zero values ​​in a given set of data. Pruning synapses and neurons in a neural network based on sparsity can reduce the amount of data to be computed by more than tenfold, with negligible loss of accuracy. Sparsity can be static or dynamic. Static sparsity is known in advance and may not change from one set of input data to another. Sparsity in weights can be considered static sparsity. For example, the weights with values ​​of zero or non-zero can be known before computation on the input data begins. Furthermore, weights with values ​​of zero or non-zero may remain unchanged from one set of input data to the next. Dynamic sparsity is the sparsity of the input feature map of the input data.

[0031] Specifically, one or more input feature maps can be generated from input data (e.g., input images). Each of these input feature maps can be combined with associated weights to perform classification processing. Each input feature map can have zero or non-zero values. The number or percentage of non-zero values ​​in a given input feature map determines its sparsity. Since each input feature map can differ from another, and the positions of zero and non-zero values ​​can change from one input feature map to another, the sparsity in input feature maps is dynamic sparsity. Because static sparsity is easier to identify and consider in machine learning operations, many mechanisms for reducing the amount of data to be computed rely solely on static sparsity. The irregularities caused by dynamic sparsity prevent many mechanisms from fully utilizing computation and reducing data.

[0032] Therefore, this disclosure provides a mechanism to transform dynamic and random sparsity into more structured or less structured sparsity to reduce the amount of data to be computed and to reduce computation time. Specifically, this disclosure provides a flexible accelerator that transforms an input feature map into multiple input sub-feature maps, such that each of the multiple sub-feature maps has a similar amount of sparsity and allows each of the multiple sub-feature maps to be processed independently while leveraging the sparsity.

[0033] See now Figure 1The diagram illustrates an exemplary block diagram of a computing system 100 according to some embodiments of the present disclosure. The computing system 100 may include a host device 105 associated with a memory device 110. The host device 105 may be configured to receive input from one or more input devices 115 and provide output to one or more output devices 120. The host device 105 may be configured to communicate with the memory device 110, the input devices 115, and the output devices 120 via appropriate interfaces 125A, 125B, and 125C, respectively. The computing system 100 may be implemented in a variety of computing devices such as computers (e.g., desktop computers, laptop computers, servers, data centers, etc.), tablet computers, personal digital assistants, mobile devices, wearable computing devices (such as smartwatches, other handheld or portable devices), or any other computing unit suitable for performing operations using the host device 105.

[0034] Input device 115 may include any of a variety of input technologies, such as a keyboard, stylus, touchscreen, mouse, trackball, keypad, microphone, voice recognition, motion recognition, remote controller, input port, one or more buttons, dial pad, joystick, and any other input peripheral associated with host device 105 that allows an external source (such as a user) to type information (e.g., data) into the host device and send instructions to the host device. Similarly, output device 120 may include a variety of output technologies, such as external storage, printer, speaker, display, microphone, LED, headphones, plotter, speech generation device, video device, GPS, and any other output peripheral configured to receive information (e.g., data) from host device 105. "Data" input to and / or output from host device 105 may include any of a variety of text data, graphic data, video data, image data, sound data, location data, combinations thereof, or other types of analog and / or digital data suitable for processing using computing system 100.

[0035] Host device 105 may include one or more central processing unit (“CPU”) cores or processors 130A-130N, which may be configured to execute instructions for running one or more applications associated with the host device. CPU cores 130A-130N are shown as non-limiting representative examples of integrated circuits capable of performing processing functions and may be replaced or combined with field-programmable gate arrays (“FPGAs”), graphics processing units (“GPUs”), custom application-specific integrated circuits (“ASICs”), etc. In some embodiments, the instructions and data required to run one or more applications may be stored within memory device 110. Host device 105 may also be configured to store the results of running one or more applications within memory device 110. Host device 105 may also include an accelerator 135. Accelerator 135 may be used to perform machine learning operations. Figure 2 Accelerator 135 is discussed in more detail. Although accelerator 135 is shown as Figure 1 In one embodiment, the accelerator is part of the host device 105, but in other embodiments, the accelerator may be separate from and communicatively coupled (e.g., via a bus or network connection) to the host device. In this case, the accelerator 135 may also be communicatively coupled to the memory device 110, becoming part of the memory device 110, or may include its own separate memory device.

[0036] To facilitate communication with memory device 110, the memory device may include or be associated with memory controller 140. Although memory controller 140 is shown as part of memory device 110, in some embodiments, memory controller may instead be part of host device 105 or another element of computing system 100 and operatively associated with memory device. Memory controller 140 may be a block of logic or circuitry configured to receive instructions from host device 105 (e.g., accelerator 135) and perform operations according to those instructions. Memory device 110 may include one or more memory modules 145 storing data and instructions. Memory module 145 may be any of a variety of memory types, including various types of volatile memory, non-volatile memory, or combinations thereof. For example, in some embodiments, one or more memory modules or portions thereof in memory module 145 may include NAND flash memory cores. In other embodiments, one or more memory modules or portions thereof in memory module 145 may include NOR flash memory cores, static random access memory (SRAM) cores, dynamic random access memory (DRAM) cores, magnetoresistive random access memory (MRAM) cores, phase-change memory (PCM) cores, resistive random access memory (ReRAM) cores, 3D XPoint memory cores, ferroelectric random access memory (FeRAM) cores, and other types of memory cores suitable for use within memory device 110. In some embodiments, one or more memory modules or portions thereof in memory module 145 may be configured as other types of storage-class memory (“SCM”). Generally, memory module 145 may include any combination of various random access memories (RAM), read-only memories (ROM), programmable ROMs (PROMs), erasable PROMs (EPROMs), electrical EPROMs (EEPROMs), hard disk drives, flash drives, memory strips, cloud storage, or main and / or auxiliary memories suitable for performing the operations described herein.

[0037] It should be understood that Figure 1Only some components of the computing system 100 are shown and described herein. However, the computing system 100 may include other components such as various batteries and power supplies, networking interfaces, routers, switches, external storage systems, controllers, etc. Generally, the computing system 100 may include any of a variety of hardware, software, and / or firmware components that are required or considered desirable in performing the functions described herein. Similarly, the host device 105, input device 115, output device 120, and memory device 110, including accelerator 135, memory controller 140, and memory module 145, may include other hardware, software, and / or firmware components that are considered necessary or desirable in performing the functions described herein. Furthermore, in some embodiments, the memory device 110 may integrate some or all of the components of the host device, including, for example, CPU cores 130A-130N and / or accelerator 135.

[0038] Now go to Figure 2 An exemplary accelerator 200 according to some embodiments of the present disclosure is shown. Accelerator 200 is similar to... Figure 1 Accelerator 135. Therefore, although not shown, accelerator 200 may be associated with a host device (e.g., host device 105) and a memory device (e.g., memory device 110). Accelerator 200 can be used to optimize machine learning operations, such as in CNNs. Although accelerator 200 is explained in relation to CNNs, in other embodiments, accelerator 200 can also be used in other types of neural network or machine learning applications. In general, accelerator 200 can be used in any type of application (whether machine learning or other applications) where a reduction in computational data and computation time is desired.

[0039] In some implementations, accelerator 200 can be used to consider dynamic and static sparsity in the input feature map and weights, respectively, and to distribute computation among various computing engines based on dynamic and static sparsity. Accelerator 200 may receive input image 205 (e.g., from host device 105) which will be subjected to one or more machine learning operations. It should be understood that although input image 205 is used as an example herein, accelerator 200 can be used to process other types of data, including video, text, and any other types of data that may benefit from being processed by the accelerator.

[0040] The input image 205 can be represented by a pixel array. Simply as an example and not intended to be limiting in any way, the input image 205 can be represented by a 1024×1024×3 pixel array. Therefore, the input image 205 is 1024 pixels high, 1024 pixels wide, and has three color depths (e.g., red, green, and blue). In some implementations, the 1024×1024×3 pixel array can be divided into three input feature maps, each representing a color and having a size of 1024×1024×1 (also simply referred to herein as 1024×1024). Furthermore, each input feature map can be represented as a matrix with multiple rows and multiple columns. Each row extends in the X direction (from left to right), and each column extends in the Y direction (from top to bottom). Each pixel of the input feature map can correspond to a cell in the matrix (e.g., formed at the intersection of a row and a column). Therefore, a 1024×1024 input feature map can be represented by a matrix with 1024 rows and 1024 columns, forming a cell of one pixel at the intersection of each row and each column.

[0041] In some embodiments, the input feature map of the input image 205 may be generated by the accelerator 200. For example, in some embodiments, the partitioning block 210 may be configured to generate the input feature map from the input image 205. In other embodiments, a feature map block (not shown) of the accelerator 200 may receive the input image 205, generate the input feature map from the input image, and send those input feature maps to the partitioning block 210. In yet another embodiment, the input feature map of the input image 205 may be generated outside the accelerator 200, and the input feature map, instead of the input image, may be input into the accelerator. The partitioning block 210 may be configured in software, hardware, firmware, or a combination thereof.

[0042] Each input feature map in the input feature map of input image 205 can be further subdivided into one or more sub-feature maps in partitioning block 210. (See below) Figure 3A and Figure 3BThe text discusses subdividing the input feature map into one or more sub-feature maps in partition block 210. Each sub-feature map is also referred to herein as an "input tensor". By segmenting the input feature map into one or more sub-feature maps, each sub-feature map in the sub-feature map can be processed independently to generate an output sub-feature map, improving performance. The output sub-feature maps from each sub-feature map in the sub-feature map can then be combined to create an output feature map. When image 205 includes multiple channels, the input feature map for each channel can be subdivided into one or more sub-feature maps, each of which can be processed independently to generate an output sub-feature map, and the output sub-feature maps from each of these one or more sub-feature maps can be combined to generate a channel output feature map. These various channel output feature maps can then be combined to generate an output feature map. Such convolutional operations involving multiple channels are referred to herein as "depthiably separable convolution". In some implementations, each sub-feature map in the input feature map and / or sub-feature map can be compressed to reduce storage space and further improve performance. The compression process used throughout this disclosure is discussed in more detail in U.S. Application No. 16 / 726,084, entitled “Flexible Accelerator For Sparse Tensors (FAST) in Machine Learning”, filed on December 23, 2019, the entire contents of which are incorporated herein by reference.

[0043] Partition block 210 may be associated with DRAM 215, which may be configured to initially store input feature maps of the input image 205, and, after subdividing these input feature maps into one or more sub-feature maps, store each of those sub-feature maps. DRAM 215 may also be used to store any intermediate outputs (e.g., output sub-feature maps, channel output feature maps, etc.) and / or output feature maps. In some embodiments, DRAM 215 may also be configured to store weight matrix 220. In some embodiments, DRAM 215 may also store various training models, schedules, and other information required to process the input image 205. Furthermore, although only DRAM 215 is shown in accelerator 200, in some embodiments, the accelerator may include additional and / or other types of memory. For simplicity, DRAM 215 is used in the specification, but other memory alternatives are contemplated for various embodiments. For example, in some embodiments, accelerator 200 may additionally or alternatively include SRAM, storage-level memory (such as MRAM, ReRAM, and / or flash memory) to store input feature maps, one or more sub-feature maps, various outputs, training models, schedules, and / or other information required to process input image 205. When such multiple memories are disposed in accelerator 200, in some embodiments, these memories may be interconnected and configured to communicate with each other. In some embodiments, DRAM 215 and any other memory in accelerator 200 may be part of a memory device (e.g., memory device 110) associated with a host device (e.g., host device 105), of which the accelerator is part. In some embodiments, one or more of DRAM 215 and any other memory in accelerator 200 may be separate from the memory device (e.g., memory device 110).

[0044] In addition to the input image 205, the accelerator 200 also receives a weight matrix 220 (e.g., from the host device 105). The weight matrix 220 may include weights or filters to be applied to each sub-feature map. The weight matrix 220 is also referred to herein as a kernel matrix or filter matrix. The weight matrix 220 may include at least one row and at least one column, forming cells at the intersections of the rows and columns. The weight matrix 220 can be used to perform convolution operations on the input image 205. In some embodiments, similar to the input feature map, the weight matrix 220 may also be subdivided into one or more sub-weight maps. In some embodiments, similar to the input feature map / sub-feature map, the weight matrix 220 and / or sub-weight maps may also be compressed. In some embodiments, the weight matrix 220 may be received by the scheduling engine 225. In other embodiments, the weight matrix 220 may be received by partitioning block 210 or by another component of the accelerator 200. Furthermore, if the weight matrix 220 is subdivided into sub-weight maps and / or compressed, in some embodiments, these processes may occur within the scheduling engine 225. In other embodiments, these processes may occur in partition block 210, in a separate partition block within accelerator 200 dedicated to weight matrix 220, and / or outside the accelerator. Weight matrix 220 and / or sub-weight maps may be stored in DRAM 215, scheduling engine 225, or any other memory of accelerator 200.

[0045] Scheduling engine 225 is configured to perform sparsity analysis, and in some embodiments, assigns each input feature map in the input feature map to a computation unit based on sparsity. As used herein, “sparseness” refers to the number or percentage of non-zero values ​​in a given input data. In some embodiments, an input feature map with more non-zero values ​​than zero values ​​is a less sparse input feature map or a dense input feature map, while an input feature map with more zero values ​​than non-zero values ​​is a sparse input feature map. In other embodiments, a sparse input feature map may be defined as having at least a certain percentage of zero values ​​(e.g., 80%), and a dense input feature map may have fewer zero values ​​than non-zero values ​​(e.g., non-zero:zero = 60:40). More generally, a sparse input feature map has more zero values ​​than a dense input feature map. Therefore, in order to perform sparsity analysis, scheduling engine 225 may determine the number or percentage of zero or non-zero values ​​in the input feature map. Furthermore, the scheduling engine 225 can dispatch or assign sparse input feature maps to the sparse tensor computation cluster 230 for processing. Therefore, the sparse tensor computation cluster 230 is configured to process input feature maps with high sparsity (e.g., the number or percentage of zero values ​​is higher than a threshold, or in other words, there are more zero values ​​than non-zero values). The sparse tensor computation cluster 230 is discussed in more detail below.

[0046] In some implementations, accelerator 200 may also include a dense tensor computation cluster ( Figure 2 (not shown in the image) and / or vector accelerators ( Figure 2 (Not shown in the diagram). If provided, the dense tensor computation cluster can be configured to process feature maps with low sparsity (e.g., dense input feature maps or where the number or percentage of zero values ​​is below a threshold, or in other words, input feature maps with more non-zero values ​​than zero values), and the vector accelerator can be used to process input feature maps that cannot be processed by the dense tensor computation cluster or the sparse tensor computation cluster 230. For example, in some embodiments that provide a dense tensor computation cluster and / or a vector accelerator, the dense tensor computation cluster and the sparse tensor computation cluster 230 can be configured to have a defined or supported list of operations that the dense tensor computation cluster and the sparse tensor computation cluster can respectively perform. If processing requires performing an operation that is not supported by either the dense tensor computation cluster or the sparse tensor computation cluster 230, the operation can be performed by the vector accelerator. In some embodiments, the vector accelerator can be an open-source vector accelerator based on RISC-V (such as LACore). In other embodiments, other suitable types of vector accelerators can be used for the vector accelerator. Dense tensor computing clusters and vector accelerators are discussed in more detail in the aforementioned U.S. patent application No. 16 / 726,084, the entire contents of which are also incorporated herein by reference.

[0047] The scheduling engine 225 may include memory 235 storing computer-readable instructions that can be executed by processor 240 to perform operations of the scheduling engine 225. Memory 235 may be part of a memory device (e.g., memory device 110) of a host device (e.g., host device 105), or may be provided from a separate memory into which accelerator 200 is part. Furthermore, the scheduling engine 225 may be implemented as software, hardware, firmware, or a combination thereof. Processor 240 may be part of or similar to CPU cores 130A-130N of host device 105, or, where the accelerator is part of memory device 110, may be part of the processor or controller of that memory device.

[0048] To process the input feature map in the sparse tensor computation cluster 230, the accelerator 200 may also include a direct memory access controller 245 configured to transfer the sub-feature map from DRAM 215 to the sparse tensor memory cluster 250 with minimal intervention from the scheduling engine 225 or the sparse tensor computation cluster. The sparse tensor memory cluster 250 is also discussed in more detail below. It should be noted that while direct memory access is used as an illustrative example, other types of memory access protocols / methods can be used, including memory access across communication buses and memory architectures.

[0049] It should be understood that Figure 2 Only some components of accelerator 200 are shown in this document. Accelerator 200 may include other or additional components that may be required or are considered desirable when performing the operations described herein.

[0050] Go to Figure 3A and Figure 3B This illustration shows an example of segmenting an input feature map into one or more sub-feature maps or input tensors in partitioning block 210 according to some embodiments of the present disclosure. Specifically, an input feature map that is a sparse input feature map and has been designated for processing in sparse tensor computation cluster 230 can be subdivided into one or more sub-feature maps. Specifically, each input feature map of each channel can be segmented into “M” sub-feature maps. “M” depends on the number of sparse tensor computation units provided within sparse tensor computation cluster 230 and / or the number of sparse tensor feature map memory units in sparse tensor memory cluster 250. For example, for five sparse tensor computation units in sparse tensor computation cluster 230, each input feature map of each channel can be segmented into five sub-feature maps. By dividing each input feature map into M sub-feature maps corresponding to M sparse tensor computation units in the sparse tensor computation cluster 230, each sub-feature map in the M sub-feature maps can be processed independently in one of the M sparse tensor computation units, thereby improving performance and reducing computation time.

[0051] In some implementations, each of the "M" sub-feature maps may be a matrix having (k-1, for example) rows and (k-1, for example) columns overlapping with adjacent sub-feature maps, where (k×k) is the size of the weight matrix 220. By overlapping the rows and columns of one sub-feature map with those of adjacent sub-feature maps, each sub-feature map can be processed independently within the sparse tensor computation cluster 230, and the output sub-feature maps of each sub-feature map can be combined together without any intra-cluster communication between the "M" sub-feature maps to create an output feature map. In other implementations, the "M" sub-feature maps may not have overlapping rows and / or columns. In the case of no overlap, it may be necessary to combine intermediate results from adjacent sub-feature maps. In some implementations, with a kernel size of 1×1, overlapping rows and / or columns of adjacent sub-feature maps may not be required.

[0052] Each input feature map can be divided into "M" sub-feature maps based on sparsity, specifically, to transform the sparsity of dynamic properties in the input feature map into near-static sparsity. Simply as an example and without intending to limit in any way, suppose the size of the input feature map for a specific channel of the input image 205 is 512×512 (e.g., 512 rows and 512 columns), and the number of sparse tensor computation units in the sparse tensor computation cluster 230 is 16 (in other words, "M" = 16). Therefore, the 512×512 input feature map can be divided into 16 sub-feature maps, where each sub-feature map is assigned to one of the 16 sparse tensor computation units in the sparse tensor computation cluster 230. To create the 16 sub-feature maps, the 512×512 input feature map can initially be divided into 16 partitions of equal or approximately equal size, such that each partition has multiple rows and multiple columns.

[0053] For example, Figure 3A An exemplary table 300 is shown after a 512×512 input feature map has been divided into 16 sub-feature maps (e.g., typically represented by cells 1 to 16). Therefore, each of the 16 partitions of the 512×512 input feature map can be 32×32 in size (e.g., 32 rows and 32 columns). Each cell in Table 300 corresponds to a 32×32 partition of the input feature map. Figure 3AThe value in parentheses within each cell indicates the percentage of non-zero values ​​in that particular cell. Thus, for example, cell 1 in Table 300 corresponds to the first partition (e.g., rows 1 to 32 and columns 1 to 32 of the input feature map) and includes 40% non-zero values ​​(and therefore 60% zero values); cell 2 corresponds to the second partition (e.g., rows 1 to 32 and columns 33 to 64) and includes 60% non-zero values ​​(and therefore 40% zero values), and so on. Cell 5 corresponds to the fifth partition (rows 33 to 64 and columns 1 to 32 of the input feature map) and includes 51% non-zero values, and so on. Figure 3A In the example, there is no overlap in the rows and columns of the 16 sub-feature maps. However, as discussed above, in some embodiments, at least some of the sub-feature maps may have rows and / or columns that overlap with adjacent sub-feature maps. In this case, the number of sub-feature maps may differ. Figure 3A 16 of the examples.

[0054] The initial partitions of table 300 can be reconfigured such that each partition includes the same or substantially similar amount or percentage of nonzero and / or zero values. In some embodiments, the percentage difference of nonzero values ​​(or zero values) can be predetermined such that any two given partitions do not have a percentage difference greater than the predetermined percentage difference in terms of nonzero values. As an example, if the predetermined percentage difference of the percentage of nonzero values ​​between any two given partitions is 3%, then the partitions can be reconfigured. Figure 3A The initial partitioning is such that each partition has approximately the same number of non-zero values, and the difference in the number of non-zero values ​​between any two partitions is no greater than 3%. Therefore, refer to Figure 3B Table 305 illustrates how to reorganize cells from Table 300. For example, cell 1 in Table 300 has 40% non-zero values, while cell 2 in Table 300 has 60% non-zero values. Therefore, the percentage difference in non-zero values ​​between cell 1 and cell 2 in Table 300 is greater than a predetermined percentage difference of 3%. Therefore, cell 1 and cell 2 can be reorganized as shown in Table 305 by assigning columns from cell 2 to cell 1. For example, three columns from cell 2 can be assigned to cell 1 such that each cell in cell 1 and cell 2 has approximately 50% non-zero values, thus making the number of non-zero values ​​in those cells equal.

[0055] After reorganization, cell 1 may have 32 rows and 35 columns, and therefore a size of 32×35, as shown in Table 305, instead of the 32×32 size shown in Table 300. Similarly, after reorganization, cell 2 may have 32 rows and 29 columns, as shown in Table 305. Therefore, cell 1 and cell 2 have different sizes, as shown in Table 305. Although only 3 columns are assigned to cell 1 from one adjacent cell, in some implementations, rows / columns may be assigned to cell 1 from multiple adjacent cells. For example, in some implementations, 2 columns may be assigned from cell 2 and 1 row may be assigned to cell 1 from cell 5. The number of rows and / or columns assigned to a particular cell from one or more cells may depend on the number of non-zero (or zero) values ​​present in the particular row and / or column and the number of non-zero (or zero) values ​​that need to be reallocated to meet a predetermined percentage difference. Similarly, other cells in Table 300 may be balanced by assigning one or more rows and / or one or more columns from one or more adjacent cells until a predetermined difference is met, as shown in Table 305.

[0056] By reorganizing the initial partitions shown in Table 300, the number of non-zero values ​​in the various initial partitions can be balanced to balance the processing in the sparse tensor computation cluster 230. Each cell in Table 305 corresponds to a sub-feature map or an input tensor, which can be assigned to a sparse tensor feature map memory unit in the sparse tensor memory cluster 250 and a sparse tensor computation unit in the sparse tensor computation cluster 230. Although Figure 3A and Figure 3B This has been interpreted as transforming the initial partition of Table 300 to the final partition of Table 305 in a single step. However, in other implementations, multiple iterations of reassigning rows / columns to adjacent cells / reassigning rows / columns from adjacent cells may be required to achieve a predetermined percentage difference. Furthermore, although a predetermined percentage difference relative to non-zero values ​​has been explained... Figure 3A and Figure 3B However, in other embodiments, different metrics may be used. For example, in some embodiments, a percentage of zero values ​​may be used instead of a percentage of non-zero values. Similarly, in some embodiments, the quantity of zero or non-zero values ​​may be used instead of a percentage of zero or non-zero values. In yet other embodiments, metrics other than "quantity" may be used. Figure 3B Each sub-feature map in the sub-feature map can be stored in DRAM 215.

[0057] refer to Figure 4 An example of a sparse tensor memory cluster 400 according to some embodiments of the present disclosure is shown. The sparse tensor memory cluster 400 is similar to... Figure 2The sparse tensor memory cluster 250 includes a sparse tensor feature map memory 405 and a sparse tensor weight memory 410. The sparse tensor feature map memory 405 is configured to store sub-feature maps received from DRAM 215 and direct memory access controller 245. The sparse tensor feature map memory 405 is also configured to store various outputs received from sparse tensor computation cluster 230. The sparse tensor feature map memory 405 includes multiple sparse tensor feature map memory cells 415A-415M. The number of these cells can depend on the specified number of sub-feature maps that may require parallel processing. Each of the multiple sparse tensor feature map memory units 415A-415M is independent of the other sparse tensor feature map memory units and can be configured to store at least one sub-feature map independently of the other sparse tensor feature map memory units.

[0058] Therefore, in some embodiments, the plurality of sparse tensor feature map memory units 415A-415M are not configured to share the data stored therein with other sparse tensor feature map memory units in the plurality of sparse tensor feature map memory units. Furthermore, each of the plurality of sparse tensor feature map memory units 415A-415M is configured to send the sub-feature map stored therein to a corresponding sparse tensor computation unit in a sparse tensor computation cluster (e.g., sparse tensor computation cluster 230). For example, in some embodiments, sparse tensor feature map memory unit #i may be configured to send the input tensor stored therein to sparse tensor computation unit #i, as discussed further below. This one-to-one correspondence between a specific sparse tensor feature map memory unit and a sparse tensor computation unit is referred to herein as a “static binding.” Therefore, in some implementations, the number of sparse tensor feature map memory units 415A-415M in the sparse tensor feature map memory 405 is the same as the number of sparse tensor computation units in the sparse tensor computation cluster (e.g., sparse tensor computation cluster 230).

[0059] Furthermore, each of the plurality of sparse tensor feature map memory units 415A-415M can be connected via bidirectional buses 420A-420M to receive sub-feature maps from DRAM 215 via direct memory access controller 245, and to send the output received from sparse tensor computation cluster 230 back to DRAM via the direct memory access controller. Similarly, each of the plurality of sparse tensor feature map memory units 415A-415M can be connected via bidirectional buses 425A-425M to an associated sparse tensor computation unit in the sparse tensor computation unit of the sparse tensor computation cluster (e.g., sparse tensor computation cluster 230), to send the sub-feature maps stored therein to the sparse tensor computation cluster and to receive the output returned from the sparse tensor computation cluster.

[0060] Therefore, for example, sparse tensor feature map memory unit 415A can receive sub-feature maps from DRAM 215 for storage via direct memory access controller 245 and bus 420A, and send the sub-feature maps to an associated sparse tensor computation unit in a sparse tensor computation unit of a sparse tensor computation cluster (e.g., sparse tensor computation cluster 230) for processing via bus 425A. Similarly, sparse tensor feature map memory unit 415A can receive outputs (e.g., results obtained from processing sub-feature maps) from a sparse tensor computation cluster (e.g., sparse tensor computation cluster 230) via bus 425A for storage, and send the outputs to DRAM 215 via direct memory access controller 245 and bus 420A. Sparse tensor feature map memory units 415B-415M can perform similar functions to sparse tensor feature map memory unit 415A.

[0061] In some implementations, each of the plurality of sparse tensor feature map memory units 415A-415M may also be configured to store index values ​​of sub-feature maps stored therein. In addition to receiving sub-feature maps, each of the plurality of sparse tensor feature map memory units 415A-415M may also receive index values ​​associated with sub-feature maps from DRAM 215. For example, if sparse tensor feature map memory unit 415A receives sub-feature map A from DRAM 215, it may also receive the index value corresponding to sub-feature map A. Sparse tensor feature map memory unit 415A may then send the index value of sub-feature map A along with sub-feature map A to a sparse tensor computing cluster (e.g., sparse tensor computing cluster 230). These index values ​​reflect the row and column numbers of a specific sub-feature map in the input feature map. For example, the index value (X, Y) refers to the row number X and column number Y of the sub-feature map in the input feature map.

[0062] The sparse tensor weight memory 410 can be configured to store weights to be applied to sub-feature maps stored in the sparse tensor feature map memory cells 415A-415M. Therefore, the sparse tensor weight memory 410 can be connected via a unidirectional bus 430 to the DRAM 215 and the direct memory access controller 245 to receive weights, and via a bus 435 to a sparse tensor computation cluster (e.g., sparse tensor computation cluster 230) to send these weights to the sparse tensor computation cluster. Since the sparse tensor weight memory 410 does not need to receive any results returned from the sparse tensor computation cluster and does not need to send any results back to the DRAM 215, buses 430 and 435 can be unidirectional buses configured to send data in a single direction. In other embodiments, buses 430 and / or 435 can be bidirectional, similar to bidirectional buses 420A-420M and / or bidirectional buses 425A-425M.

[0063] Go to Figure 5 An example of a sparse tensor memory cluster 500 according to some embodiments of the present disclosure is shown. The sparse tensor memory cluster 500 is similar to... Figure 2The sparse tensor memory cluster 250. The sparse tensor memory cluster 500 is also substantially similar to the sparse tensor memory cluster 400. For example, similar to the sparse tensor memory cluster 400, the sparse tensor memory cluster 500 includes a sparse tensor feature map memory 505 and a sparse tensor weight memory 510. Also similar to the sparse tensor feature map memory 405, the sparse tensor feature map memory 505 includes a plurality of sparse tensor feature map memory cells 515A-515M connected to the DRAM 215 and the direct memory access controller 245 via bidirectional buses 520A-520M. However, unlike the sparse tensor feature map memory 405 (where each of the plurality of sparse tensor feature map memory units 515A-515M is independent, does not share the data stored therein with other sparse tensor feature map memory units in the plurality of sparse tensor feature map memory units, and sends the data stored therein to the corresponding sparse tensor computation unit in the sparse tensor computation unit), the plurality of sparse tensor feature map memory units 515A-515M of the sparse tensor feature map memory 505 are interconnected with each other and interconnected to other sparse tensor computation units via memory interconnect 525.

[0064] Furthermore, in some embodiments, the memory interconnect 525 can be configured to override the static assemblage discussed above. For example, in some embodiments, the memory interconnect 525 can enable the sparse tensor feature map memory cell #i to communicate with the sparse tensor computation cells #1-M ("M" is the number of sparse tensor computation cells in the associated sparse tensor computation cluster) depending on the configuration of the memory interconnect. In some embodiments, the memory interconnect 525 can be a 2×2 switch that enables the sparse tensor feature map memory cell #i to communicate with either sparse tensor computation cell #i or sparse tensor computation cell #i+1. In other embodiments, the memory interconnect 525 can be a multi-level interconnect, such as a mesh network or a Benes network, that allows the sparse tensor feature map memory cell #i to communicate with each of the sparse tensor computation cells in the sparse tensor computation cells #1-M. In other embodiments, memory interconnect 525 may be configured in other ways to allow sparse tensor feature map memory cell #i to communicate with one or more sparse tensor computation cells other than sparse tensor computation cell #i. Similarly, in some embodiments, memory interconnect 525 may enable a specific sparse tensor feature map memory cell among a plurality of sparse tensor feature map memory cells 515A-515M to interconnect with one or more other sparse tensor feature map memory cells among the plurality of sparse tensor feature map memory cells. For example, depending on the configuration of memory interconnect 525, sparse tensor feature map memory cell #i may interconnect with one or more sparse tensor feature map memory cells #(i+1)-M.

[0065] Each of the multiple sparse tensor feature map memory units 515A-515M can be connected to memory interconnect 525 via bidirectional buses 530A-530M. Therefore, each of the multiple sparse tensor feature map memory units 515A-515M can be configured to send sub-feature maps (and corresponding index values) stored therein to memory interconnect 525, and receive sub-feature maps (e.g., stored in another sparse tensor feature map memory unit) or output them from memory interconnect via corresponding bidirectional buses in its bidirectional buses 530A-530M. Similarly, memory interconnect 525 can be connected to a sparse tensor computation cluster (e.g., sparse tensor computation cluster 230) via bidirectional buses 535A-535M to send sub-feature maps (and index values) to the sparse tensor computation cluster and receive outputs from the sparse tensor computation cluster. By using the memory interconnect 525, the flexibility of storing information in multiple sparse tensor feature map memory cells 515A-515M can be increased, and the static combination of sparse tensor memory cluster 400 can be overridden.

[0066] The sparse tensor weight memory 510 is configured similarly to the sparse tensor weight memory 410. Therefore, the sparse tensor weight memory 510 can be configured to store weights to be applied to sub-feature maps stored in the sparse tensor feature map memory cells 515A-515M. Furthermore, the sparse tensor weight memory 510 can be connected via a unidirectional bus 540 to the DRAM 215 and the direct memory access controller 245 to receive weights, and via a bus 545 to a sparse tensor computing cluster (e.g., sparse tensor computing cluster 230) to send these weights to the sparse tensor computing cluster. In other embodiments, bus 540 and / or bus 545 can be bidirectional.

[0067] refer to Figure 6An exemplary sparse tensor feature map memory cell 600 according to some embodiments of the present disclosure is shown. The sparse tensor feature map memory cell 600 is similar to each of a plurality of sparse tensor feature map memory cells 415A-415M and a plurality of sparse tensor feature map memory cells 515A-515M. The sparse tensor feature map memory cell 600 includes a write switch 605, a read switch 610, a first set of buffers 615, and a second set of buffers 620. The write switch 605 is configured to write sub-feature maps received from DRAM 215 (or from another sparse tensor feature map memory cell in the case of an interconnect) to the first set of buffers 615 and / or the second set of buffers 620. The write switch 605 is also configured to write outputs (e.g., output sub-feature maps) received from a sparse tensor computing cluster (e.g., sparse tensor computing cluster 230) to the first set of buffers 615 and / or the second set of buffers 620. In some implementations, write switch 605 may be a 2×2 switch configured for double-buffered control to receive data from two sources and write that data to two sets of buffers (e.g., a first set of buffers 615 and a second set of buffers 620). In other implementations, write switch 605 may be configured in other ways.

[0068] The read switch 610 can be configured to read data stored in the first buffer 615 and the second buffer 620. For example, the read switch 610 can read data written by the write switch 605 in the first buffer 615 and / or the second buffer 620 to send the read data (via the direct memory access controller 245) to the DRAM 215. Similarly, the read switch 610 can read data written by the write switch 605 in the first buffer 615 and / or the second buffer 620 to send the read data to a sparse tensor computation cluster (and particularly, a sparse tensor computation unit of the sparse tensor computation cluster) associated with the sparse tensor feature map memory unit 600. For example, the write switch 605 can receive a sub-feature map (and corresponding index values) from the DRAM 215 and store the sub-feature map (and index values) in the first buffer 615 and / or the second buffer 620. Read switch 610 can then read the sub-feature map (and index values) from the first set of buffers 615 and / or the second set of buffers 620, and send the read data to the sparse tensor computation cluster. Similarly, write switch 605 can receive the output sub-feature map from the sparse tensor computation cluster and write the output sub-feature map into the first set of buffers 615 and / or the second set of buffers 620. Read switch 610 can read the output from the first set of buffers 615 and / or the second set of buffers 620, and transfer the output tensor to DRAM 215.

[0069] In some embodiments, the read switch 610 may also be a 2×2 switch configured for double-buffered control to read data from two sets of buffers (e.g., a first set of buffers 615 and a second set of buffers 620). In other embodiments, the read switch 610 may be a 1×1 switch configured to read data from a single set of buffers, or the read switch may be configured to read data from more than two sets of buffers.

[0070] In some embodiments, each of the first set of buffers 615 and the second set of buffers 620 may include two buffers. For example, in some embodiments, the first set of buffers 615 may include a first value buffer 625 and a first index buffer 630. Similarly, in some embodiments, the second set of buffers 620 may include a second value buffer 635 and a second index buffer 640. Although only two sets of buffers (e.g., the first set of buffers 615 and the second set of buffers 620) are shown in the sparse tensor feature map memory cell 600, in other embodiments, a single set of buffers or more than two sets of buffers may be provided depending on the configuration of the write switch 605 and / or the read switch 610. Similarly, although each of the first set of buffers 615 and the second set of buffers 620 is shown as having two buffers each, in other embodiments, either or both sets of buffers in the first set of buffers and the second set of buffers may each include more than two buffers or may include a single buffer.

[0071] The first value buffer 625 and the second value buffer 635 can be configured to store the data values ​​of the sub-feature map or the output sub-feature map, while the first index buffer 630 and the second index buffer 640 can be configured to store the index values ​​of the sub-feature map or the output sub-feature map. For example, in some embodiments, the data value of sub-feature map A can be stored in the first value buffer 625, and the index value of the sub-feature map can be stored in the first index buffer 630. In other embodiments, the data value of a specific sub-feature map (or the output sub-feature map) can be stored in one set of buffers in the first set of buffers 615 or the second set of buffers 620, and the index value of the specific sub-feature map (or the output sub-feature map) can be stored in the other set of buffers in the first set of buffers or the second set of buffers. Furthermore, in some embodiments, the first set of buffers 615 can be designated to store the data values ​​and index values ​​of the sub-feature map, while the second set of buffers can be configured to store the data values ​​and index values ​​of the output sub-feature map. In other embodiments, each of the first set of buffers 615 and the second set of buffers 620 may store both the sub-feature map and the output sub-feature map (and their corresponding index values). Therefore, each sub-feature map and each output sub-feature map may be associated with two buffers: a value buffer (e.g., the first value buffer 625, the second value buffer 635) for storing the data values ​​of the sub-feature map or the output sub-feature map, and an index buffer (e.g., the first index buffer 630, the second index buffer 640) for storing the index values ​​of the sub-feature map or the output sub-feature map.

[0072] Furthermore, although the data values ​​and index values ​​of a specific sub-feature map or output sub-feature map are shown as being stored in separate buffers (e.g., first value buffer 625, second value buffer 635, first index buffer 630, second index buffer 640), in some embodiments, the data values ​​and index values ​​of a specific sub-feature map or output sub-feature map may be stored within a single buffer. In other words, in some embodiments, first value buffer 625 and first index buffer 630 may be combined to form a single buffer. Similarly, in some embodiments, second value buffer 635 and second index buffer 640 may be combined to form a single buffer.

[0073] Each buffer in the first group of buffers 615 and the second group of buffers 620 can be an SRAM memory configured to read / write a register file, a first-in-first-out data structure, a set of registers, etc. By using SRAM memory for the buffers in the first group of buffers 615 and the second group of buffers 620, complex and more expensive cache structures can be avoided. In other embodiments, one or more buffers in the first group of buffers 615 and / or the second group of buffers 620 can be other types of memory. Furthermore, each buffer in the first group of buffers 615 and the second group of buffers 620 can be configured to have a specific size capable of accommodating data values ​​and index values ​​of at least one sub-feature map or at least one output sub-feature map.

[0074] Now for reference Figure 7 An exemplary sparse tensor weight memory 700 according to some embodiments of the present disclosure is shown. The sparse tensor weight memory 700 is similar to sparse tensor weight memories 410 and 510. The sparse tensor weight memory 700 includes a first buffer 705 configured to store weight values ​​and a second buffer 710 for storing index values ​​of the weight values ​​from a weight matrix 220. Thus, the first buffer 705 is similar to a first value buffer 625 and a second value buffer 635, while the second buffer 710 is similar to a first index buffer 630 and a second index buffer 640. The first buffer 705 and the second buffer 710 can receive and store weight values / index values ​​from DRAM 215 and send those values ​​to a sparse tensor computing cluster (e.g., sparse tensor computing cluster 230). In some embodiments, more than one buffer may be used for storing weight values ​​and / or more than one buffer for storing index values ​​of the weight matrix.

[0075] Go to Figure 8An exemplary sparse tensor computation cluster 800 according to some embodiments of the present disclosure is shown. The sparse tensor computation cluster 800 is similar to the sparse tensor computation cluster 230. The sparse tensor computation cluster 800 includes a plurality of sparse tensor computation units 805A-805M. The number of the plurality of sparse tensor computation units 805A-805M may depend on a specified number of sub-feature maps that may need to be processed in parallel. For example, to process five sub-feature maps in parallel, five sparse tensor computation units may be provided, wherein each sparse tensor computation unit is configured to process one sub-feature map at a time. In some embodiments, the number of the plurality of sparse tensor computation units 805A-805M is the same as the number of the plurality of sparse tensor feature map memory units 415A-415M or 515A-515M, wherein sparse tensor computation unit #i is associated with sparse tensor feature map memory unit #i. In other embodiments, different numbers of multiple sparse tensor computation units 805A-805M and multiple sparse tensor feature map memory units 415A-415M or 515A-515M can be used.

[0076] Furthermore, in some embodiments, each of the plurality of sparse tensor computation units 805A-805M can process data independently of the other sparse tensor computation units. Each of the plurality of sparse tensor computation units 805A-805M receives a sub-feature map (and corresponding index value) from the plurality of sparse tensor feature map memory units 415A-415M or 515A-515M via a bidirectional bus 810A-810M. The bidirectional bus 810A-810M can also be used to send the output sub-feature map back to the plurality of sparse tensor feature map memory units 415A-415M or 515A-515M.

[0077] For example, if a sparse tensor memory cluster 400 with static binding is used, in some embodiments, the sparse tensor computation unit #i can be configured to receive sub-feature maps (and corresponding index values) stored in the sparse tensor feature map memory unit #i via a bidirectional bus #i. In such embodiments, the sparse tensor computation unit #i can also be configured to send an output sub-feature map to the sparse tensor feature map memory unit #i via the bidirectional bus #i. Thus, there is a one-to-one correspondence between specific sparse tensor computation units and sparse tensor feature map memory units. For example, during static binding, the sparse tensor computation unit 805A can receive a sub-feature map (and corresponding index values) from the sparse tensor feature map memory unit 415A via a bidirectional bus 810A, and can send the resulting output sub-feature map back to the sparse tensor feature map memory unit 415A via the bidirectional bus 810A.

[0078] In other embodiments, if a sparse tensor memory cluster 500 is used, the sparse tensor computation unit #i can still be associated with the sparse tensor feature map memory unit #i. However, depending on the configuration of the memory interconnect 525, the sparse tensor computation unit #i can receive sub-feature maps from the sparse tensor feature map memory units #(i+1)-M via bidirectional buses 810A-810M. Furthermore, depending on the configuration of the memory interconnect 525, in addition to the sparse tensor feature map memory unit #i, the sparse tensor computation unit #i may also be able to send output sub-feature maps to the sparse tensor feature map memory units #(i+1)-M.

[0079] In addition to the sub-feature map, each of the plurality of sparse tensor computation units 805A-805M also receives weight values ​​(and corresponding index values) from the sparse tensor weight memory 410 or sparse tensor weight memory 510 via a unidirectional bus 815A-815M. In some embodiments, the same weight may be transferred to each or at least a group of the plurality of sparse tensor computation units 805A-805M via the unidirectional bus 815A-815M. In other embodiments, different weights may be transferred to each or at least a group of the plurality of sparse tensor computation units 805A-805M via the unidirectional bus 815A-815M. Furthermore, in some embodiments, a single weight may be transferred to the plurality of sparse tensor computation units 805A-805M at a time, while in other embodiments, more than one weight may be transferred simultaneously to one or more of the plurality of sparse tensor computation units.

[0080] The output sub-feature map obtained by processing the sub-feature map can be transmitted back to the corresponding sparse tensor feature map memory unit in the sparse tensor feature map memory unit. For example, in some embodiments, the sparse tensor computation unit 805A may receive a sub-feature map from the sparse tensor feature map memory unit 415A or 515A, process the sub-feature map to obtain an output sub-feature map, and send the output sub-feature map back to the sparse tensor feature map memory unit 415A or 515A. Then, the sparse tensor feature map memory unit 415A or 515A may, based on configuration, send the output sub-feature map to DRAM 215, to another sparse tensor feature map memory unit, and / or to another sparse tensor computation unit.

[0081] Go to Figure 9 An exemplary sparse tensor computation cluster 900 according to some embodiments of the present disclosure is shown. The sparse tensor computation cluster 900 is similar to the sparse tensor computation cluster 230. The sparse tensor computation cluster 900 is also substantially similar to the sparse tensor computation cluster 800. For example, similar to the sparse tensor computation cluster 800, the sparse tensor computation cluster 900 includes a plurality of sparse tensor computation units 905A-905M, which are connected via bidirectional buses 910A-910M to at least one of the sparse tensor feature map memory units 415A-415M or 515A-515M, as discussed above. Also similar to the sparse tensor computation cluster 800, each of the plurality of sparse tensor computation units 905A-905M is connected via a unidirectional bus 915A-915M to a sparse tensor weight memory 410 or 510 to receive weights.

[0082] However, unlike the sparse tensor computation cluster 800 (where each of the plurality of sparse tensor computation units 805A-805M is independent and does not share the data being processed therein), the plurality of sparse tensor computation units 905A-905M of the sparse tensor computation cluster 900 are interconnected via an interconnector 920. The interconnector 920 can be configured to override the static assemblage discussed above. Therefore, in some embodiments, the interconnector 920 enables sparse tensor computation unit #i to communicate with other sparse tensor computation units #1-M, depending on the configuration of the interconnector. For example, in some embodiments, the interconnector 920 can be a 2×2 switch that enables sparse tensor computation unit #i to communicate with sparse tensor computation unit #i+1. In other embodiments, the interconnector 920 may be a multi-level interconnect, such as a mesh network or a Benes network, that allows each sparse tensor computation unit #i to communicate with each other sparse tensor computation units #1-M.

[0083] Each of the multiple sparse tensor computation units 905A-905M can be connected to the interconnect 920 via a bidirectional bus 925A-925M. Therefore, each of the multiple sparse tensor computation units 905A-905M can be configured to directly send the output sub-feature map generated by processing a specific sub-feature map to another sparse tensor computation unit among the multiple sparse tensor computation units via the interconnect 920 and the bidirectional bus 925A-925M, instead of first sending those results to the sparse tensor memory cluster 400 or 500. The use of the interconnect 925 increases the flexibility of dispatching and processing sub-feature maps.

[0084] Go to Figure 10 An exemplary block diagram of a sparse tensor computation unit 1000 according to some embodiments of the present disclosure is shown. The sparse tensor computation unit 1000 is similar to one of a plurality of sparse tensor computation units 805A-805M or 905A-905M. The sparse tensor computation unit 1000 is configured to perform various machine learning operations, such as multiplication, addition, etc., that may be performed during convolution operations in a CNN. Therefore, the sparse tensor computation unit 1000 receives sub-feature maps (and corresponding index values) (as discussed above) from a sparse tensor memory cluster 400 or 500, or receives an output sub-feature map from another sparse tensor computation unit. The sparse tensor computation unit 1000 also receives weight values ​​from a sparse tensor weight memory 410 or 510.

[0085] The sparse tensor computation unit 1000 includes a plurality of multipliers 1005A-1005P, each of which is configured to multiply the data values ​​of the sub-feature map by the weight values ​​of the weight matrix 220. In some embodiments, the number of the plurality of multipliers 1005A-1005P may depend on the number of cells in the sub-feature map. For example, for a 2×2 sub-feature map with a total of four cells spanning two rows and two columns, in some embodiments, the number of the plurality of multipliers 1005A-1005P may be four so that the data values ​​in each cell can be processed independently. In other embodiments, the number of the plurality of multipliers 1005A-1005P may be greater than or less than the number of cells in the sub-feature map.

[0086] For example, in some implementations, the number of multiple multipliers 1005A-1005P may depend on the number of cells with non-zero values ​​in the sub-feature map. For example, in the 2×2 sub-feature map with four cells described above, if only three cells have non-zero values, the number of multiple multipliers 1005A-1005P may be three to process the three non-zero values ​​in parallel. Alternatively, the number of multiple multipliers 1005A-1005P may still be four; however, only three of the four multipliers may participate in / be used to process the three non-zero values. In other implementations, the number of multiple multipliers 1005A-1005P may be two, and two of the three non-zero values ​​may be processed in parallel in the first round, and the third non-zero value may be processed in one of the two multipliers in the second round after the first round.

[0087] In some implementations, all P multiplier units can be used in each clock cycle. For example, if the number of multiple multipliers 1005A-1005P is 4, and there are 3 non-zero values ​​(d1, d2, d3) in a 2×2 cell (to which weights w1 and w2 will be applied), then at clock cycle 1, all 4 multipliers can be utilized as follows: d1*w1, d2*w1, d3*w1, and d1*w2. Generally, if P is the total number of data values ​​in the sub-feature map processed by the sparse tensor computation cluster 1000, and Q is the number of non-zero values ​​in the sub-feature map, then in some implementations (e.g., in the CONV2 layer), Q multipliers can participate, and a unique weight of max(1, ceiling(PQ) / Q) can be transferred to the Q multipliers to ensure full utilization of the multiple multipliers 1005A-1005P. For example, in the case of a single unique weight value, that unique weight value can be transferred to each of the Q multipliers. With two unique weight values, the first weight value can be transferred to Q multipliers, and the second weight value can be transferred to the remaining number of multipliers. In the case of 1×1 CONV and FC layers, P weight values ​​can be transferred to P multipliers. Therefore, in some implementations, only the non-zero data values ​​of the sub-feature map are input into the multiple multipliers 1005A-1005P. Since the product with zero data values ​​is zero, it is not necessary to process any zero values ​​in the input sub-feature map through the multiple multipliers 1005A-1005P, thus saving computational resources and time. Conversely, in some implementations, after the computation results of filling the output feature map with non-zero values, the remaining index values ​​can be filled with zero values.

[0088] In some implementations, the number of multiple multipliers 1005A-1005P may depend on the size of the weight matrix 220 (also referred to herein as the kernel matrix or filter matrix). The weight matrix 220 may also include multiple cells, as discussed above. For example, the weight matrix 220 may have two rows and two columns forming four cells. Therefore, the number of multiple multipliers 1005A-1005P provided or involved can be four to process four data values ​​in parallel. Thus, the number of multiple multipliers 1005A-1005P may depend on a variety of factors.

[0089] In some implementations, only non-zero weight values ​​may be input into the multiple multipliers 1005A-1005P. Therefore, for example, if the weight matrix 220 has four cells and only three of those cells have non-zero values, only three weight values ​​may be input into the multiple multipliers 1005A-1005P. In some implementations, the number of non-zero weight values ​​may not affect the number of multiple multipliers 1005A-1005P, but may affect the number of iterations required to process the sub-feature map, as discussed below. Similar to zero data values, the appropriate index value for the position in the output sub-feature map where the result of multiplying by a zero weight value will be calculated. Those index values ​​can then be filled with zero values.

[0090] Furthermore, in some embodiments, each sparse tensor computation unit (e.g., multiple sparse tensor computation units 805A-805M, 905A-905M) in a sparse tensor computation cluster (e.g., multiple multipliers 1005A-1005P) may have the same number of multipliers, while in other embodiments, the number of multipliers in one or more sparse tensor computation units of a sparse tensor computation cluster may be different from that in other sparse tensor computation units in the same sparse tensor computation cluster.

[0091] Furthermore, the size of each of the multiple multipliers 1005A-1005P can be determined based on the dimensions of the data values ​​and weight values ​​to be processed therein. In some embodiments, each of the multiple multipliers 1005A-1005P can be an electronic circuit configured to multiply two binary numbers. Generally, each of the multiple multipliers 1005A-1005P can be implemented using software, hardware, firmware, or a combination thereof in any of a variety of ways.

[0092] Therefore, each of the multiple multipliers 1005A-1005P receives data values ​​1010A-1010P from the sub-feature map. For example, and in combination Figure 10 refer to Figure 11The sparse tensor computation unit 1000 is explained relative to the exemplary 5×5 input feature map 1100. It should be understood that the input feature map 1100 is merely an example and is not intended to be limiting in any way. This disclosure can be used to process input feature maps of any size and containing any data values. The input feature map 1100 can be generated from the input image 205. The input feature map 1100 may include multiple cells, each formed at the intersection of a row (which extends in the X direction 1105) and a column (which extends in the Y direction 1110). Each of these multiple cells in the input feature map 1100 includes data values ​​that will be processed in the sparse tensor computation unit 1000.

[0093] In some implementations, the four sides of the input feature map 1100 can be padded with zero values ​​to ensure that the output feature map has the same size as the input feature map. For example, a row of zero values ​​can be added above the first row and below the last row of the input feature map 1100, and a column of zero values ​​can be added to the left-hand side of the first column and the right-hand side of the last column of the input feature map for padding. By padding the input feature map 1100 with zero values, a padded feature map 1115 with a size of 7×7 is obtained. In some implementations, the partition block 210 can perform the padding. In other implementations, the padding can be added by another component of the accelerator 200. Without zero padding, the output feature map may have a different size than the input feature map.

[0094] Furthermore, in some embodiments, sub-feature maps can be created from the filled feature map 1115, and weight values ​​from kernel matrix 1120 can be applied to those sub-feature maps. It should be understood that kernel matrix 1120 is merely an example and is not intended to be limiting in any way. Kernel matrix 1120 may assume other dimensions (e.g., the number of rows and columns may differ from those shown), and the values ​​within the kernel matrix may also vary. Kernel matrix 1120 can be considered to have a kernel size k. In some embodiments, in a square kernel matrix (e.g., kernel matrix 1120) with the same number of rows and columns, the kernel size k is equal to the number of rows or columns in the kernel matrix. In other embodiments, the kernel size k of a square or non-square kernel matrix can be considered as the input parameters that can be determined / optimized using an external training process involving backpropagation of supervised examples and error gradients. Therefore, for kernel matrix 1120 with two rows and two columns, the following description assumes a kernel size k of two.

[0095] In some implementations, the filled feature map 1115 may be segmented into sub-feature maps within the partition block 210 or other components of the accelerator 200, as described above. Figure 3A and Figure 3B As discussed in [the document]. For example, and as [the document discusses]. Figure 11 As shown, the filled feature map 1115 (as discussed above) can be segmented to form multiple sub-feature maps 1125A-1125I. Since the last row and last column of the filled feature map 1115 only include zero values, those values ​​have no effect on the output value and therefore do not need to be processed. Each sub-feature map in the multiple sub-feature maps 1125A-1125I can be processed in parallel in different sparse tensor computation units (e.g., sparse tensor computation unit 1000). For example, in some embodiments, sub-feature map 1125A can be processed in a first sparse tensor computation unit, sub-feature map 1125B can be processed in a second sparse tensor computation unit, and so on.

[0096] In addition, Figure 11 In the multiple sub-feature mapping maps 1125A-1125I, each sub-feature mapping map comprises two rows and two columns. Although each sub-feature mapping map in the multiple sub-feature mapping maps 1125A-1125I is similar to... Figure 11 Other sub-feature maps in the matrix have the same size, but it should be understood that these sub-feature maps can have different sizes, as discussed above. Furthermore, although each of the multiple sub-feature maps 1125A-1125I has the same number of rows and columns as the kernel matrix 1120, in some embodiments, one or more of the multiple sub-feature maps can have a different number of rows and / or columns than the kernel matrix. Depending on the number of the multiple multipliers 1005A-1005P in the sparse tensor computation unit 1000, multiple data values ​​in each sub-feature map of the multiple sub-feature maps 1125A-1125I can be processed in parallel. For example, if sub-feature map 1125A is processed in the sparse tensor computation unit 1000, and assuming that the multiple multipliers 1005A-1005P include at least four multipliers, the data values ​​in each of the four cells of that sub-feature map can be processed in parallel.

[0097] Therefore, to process the sub-feature map 1125A, data values ​​from this sub-feature map can be input into multiple multipliers 1005A-1005P. For example, the data value "0" with index (1,1) (e.g., row 1, column 1) of the sub-feature map 1125A can be loaded into multiplier 1005A, the data value "0" with index (1,2) (e.g., row 1, column 2) can be loaded into multiplier 1005B, the data value "1" with index (2,1) (e.g., row 2, column 1) can be loaded into multiplier 1005C, and the data value "5" with index (2,2) (e.g., row 2, column 2) can be loaded into multiplier 1005P. In some embodiments, the corresponding index values ​​of the data values ​​can also be input into the corresponding multipliers in the multiple multipliers 1005A-1005P. In this document, zero data values ​​are loaded into multiple multipliers 1005A-1005P only for illustrative purposes. In other embodiments, only non-zero data values ​​may be loaded into multiple multipliers 1005A-1005P.

[0098] In addition to the data values, each of the multiple multipliers 1005A-1005P also receives weight values ​​1015A-1015P from the sparse tensor weight memory 410 or 510. The weight values ​​1015A-1015P can be values ​​from cells of the kernel matrix 1120. In some embodiments, the sparse tensor computation unit 1000 can be configured to process one unique weight at a time. In this case, a single weight value can be broadcast to each of the multiple multipliers 1005A-1005P at a time. For example, in the first iteration of processing the sub-feature map 1125A, a first weight value from the kernel matrix 1120 can be transferred to each of the multipliers 1005A, 1005B, 1005C, and 1005P (hereinafter collectively referred to as the multiple multipliers 1005A-1005P). Therefore, the same weight value is broadcast to each of the multiple multipliers 1005A-1005P.

[0099] After processing the sub-feature map 1125A using the first weight value, a second weight value from the kernel matrix 1120 can be transferred to each of the multiple multipliers 1005A-1005P in the second iteration. After processing the sub-feature map 1125A using the second weight value, a third weight value from the kernel matrix 1120 can be transferred to each of the multiple multipliers 1005A-1005P in the third iteration, and after processing the sub-feature map using the third weight value, a fourth weight value from the kernel matrix can be transferred to each of the multiple multipliers in the fourth iteration. Therefore, processing the sub-feature map 1125A may require four iterations. During each iteration, the input weight value is multiplied by each data value in the sub-feature map 1125A. Furthermore, depending on the number of multiple multipliers 1005A-1005P, each iteration may include one or more rounds. Specifically, if the number of multiple multipliers 1005A-1005P is sufficient to process all data values ​​of the sub-feature map in parallel, then each iteration may include a single round. On the other hand, if the number of multiple multipliers 1005A-1005P is less than the number of data values ​​in the sub-feature map, then each iteration may include multiple rounds. After completing four iterations using the sub-feature map 1125A, the output of the sparse tensor computation unit 1000 may be an output sub-feature map corresponding to the sub-feature map 1125A.

[0100] Furthermore, in some embodiments, the order in which the weight values ​​from kernel matrix 1120 are transferred to the multiple multipliers 1005A-1005P can be predetermined. For example, in some embodiments, the weight value with index (1,1) from kernel matrix 1120 can be loaded into each of the multiple multipliers 1005A-1005P in the first iteration. As mentioned above, this index value reflects the row and column number of a specific cell in kernel matrix 1120. Therefore, the index value (1,1) corresponds to row 1 and column 1 of kernel matrix 1120. The weight value corresponding to the index value (1,1) in kernel matrix 1120 is "0". Therefore, in the first iteration, the weight value "0" is loaded into each of the multiple multipliers 1005A-1005P. Similar to data values, describing a zero weight value as being input into the multiple multipliers 1005A-1005P is merely for illustrative purposes. In other implementations, only non-zero weight values ​​can be input into the multiple multipliers 1005A-1005P. In the second iteration, the weight value at index (1,2) (e.g., weight value "1") is broadcast to each of the multiple multipliers 1005A-1005P. In the third iteration, the weight value corresponding to index (2,2) (e.g., with weight value "0") is loaded into each of the multiple multipliers 1005A-1005P, and in the fourth iteration, the weight value "-1" corresponding to index (1,1) is broadcast to each of the multiple multipliers. In other implementations, the weight values ​​can be input into the multiple multipliers 1005A-1005P in different orders in various iterations.

[0101] Therefore, in the first iteration of processing the sub-feature map 1125A, each of the multiple multipliers 1005A-1005P receives a data value from the sub-feature map and a weight value "0" corresponding to the index (1,1) in the kernel matrix 1120. Each of the multiple multipliers 1005A-1005P computes the product (e.g., z = w * x) between its corresponding data value (e.g., data values ​​1010A-1010P) and the weight value (e.g., weight values ​​1015A-1015P) to obtain the product. For example, multiplier 1005A can multiply the data value 1010A by the weight value 1015A to obtain the product value 1020A. Similarly, multiplier 1005B can multiply data value 1010B with weight value 1015B to obtain product value 1020B, multiplier 1005C can multiply data value 1010C with weight value 1015C to obtain product value 1020C, and multiplier 1005P can multiply data value 1010P with weight value 1015P to obtain product value 1020P. The product values ​​1020A-1020P can be represented as a product matrix such that each product value in 1020A-1020P has the same index value as the data value input into the corresponding multiplier in the plurality of multipliers 1005A-1005P. For example, since a data value with index value (1, 1) is input into multiplier 1005A, product value 1020A also has the index value (1, 1) in the product matrix. Therefore, the product matrix based on the product value 1020A-1020P can look like this: 0 0 0 0

[0104] Each product value in product values ​​1020A-1202P is input into the corresponding accumulator 1025A-1025P. Each accumulator in accumulators 1025A-1025P may include a register (or other type of memory) configured to receive and temporarily store the corresponding product value in product values ​​1020A-1020P. In some embodiments, each accumulator in accumulators 1025A-1025P may also include a computational element (e.g., an adder element) to perform the calculations discussed below. Although in Figure 10The diagram shows P individual accumulators in accumulators 1025A-1025P, but in some embodiments, one or more of the P accumulators may be combined to form a larger accumulator that receives the product values ​​1020A-1020P. Additionally, in some embodiments, the size of each accumulator in accumulators 1025A-1025P can be calculated using the following formula: (2k-1) × (2k-1), where k is the kernel size of kernel matrix 1120. Since the kernel size k of kernel matrix 1120 is 2 in the current example, each accumulator in accumulators 1025A-1025P can be sized to store at least a 3×3 matrix.

[0105] Furthermore, after transferring the product values ​​1020A-1020P to the accumulators 1025A-1025P, the second iteration of processing the sub-feature map 1125A can begin in the multiple multipliers 1005A-1005P. Therefore, in the second iteration, the weight values ​​corresponding to the index values ​​(1,2) of the kernel matrix 1120 can be transferred to each of the multiple multipliers 1005A-1005P. Since the data values ​​of the sub-feature map 1125A have already been processed in the multiple multipliers 1005A-1005P since the first iteration, it is unnecessary to input those data values ​​again.

[0106] Furthermore, upon receiving the product values ​​1020A-1020P, each accumulator in accumulators 1025A-1025P can process those values ​​based on the values ​​received from register group 1030. Register group 1030 is configured to store the output sub-feature map resulting from the convolution of sub-feature map 1125A with kernel matrix 1120. Register group 1030 can be a set of registers, flip-flops, or other memory units. Although register group 1030 is explained herein from the perspective of registers, in other embodiments, flip-flops or other types of memory units can be used in the register group. In some embodiments, register group 1030 can be part of one or more accumulators in accumulators 1125A-1125P. Furthermore, in some embodiments, register group 1030 can include a plurality of row registers and a plurality of column registers connected together to form one or more shift registers. In some embodiments, the plurality of row registers can be connected together to form a shift register such that the value stored therein can be shifted to the right or left by at least one position. Similarly, in some implementations, the multiple column registers can be connected together to form a shift register, so that the value stored therein can be shifted up or down by at least one position.

[0107] Furthermore, the size of register set 1030 can be based on the size of the filled feature map 1115 (or the size of the output feature map). Therefore, for a filled feature map 1115 of size 7×7, register set 1030 can be initialized to have a size of 7×7. In other words, register set 1030 can include 49 registers, such that 7 registers in each row are connected together to form a shift register and 7 registers in each column are connected together to form a shift register. Additionally, in some embodiments, register set 1030 can be initialized to have only zero values ​​at the start of the first iteration, such as... Figure 11 The register group 1130 is shown in the diagram. Additionally, in some embodiments, at the start of the first iteration, a portion of register group 1130 may be loaded into each of the accumulators 1025A-1025P.

[0108] For example, each accumulator in accumulators 1025A-1025P may include a register 1135 with a size of (2k-1) × (2k-1) or 3×3, as discussed above. When register set 1130 is initialized to have only 0 values ​​at the start of the first iteration, register 1135 of each accumulator in accumulators 1025A-1025P can be loaded with the value from the 3×3 portion of register set 1130. The 3×3 portion from register set 1130 copied to register 1135 can be determined based on the following formula:

[0109] Register = Register set (i: i + acc_length - 1, j: j + acc_length - 1)

[0110] In the above formula, the first term corresponds to the row number of register group 1130, and the second term corresponds to the column number of register group. Furthermore, in the above formula, "i" is the start of the row number, and "j" is the start of the column number of sub-feature map 1125A in the filled feature map 1115, and the acc_ length is the size of register 1135. In other words, (i, j) is the index value of sub-feature map 1125A. For example, since the register is 3×3, the size of register 1135 in the above example is 3. The size of register 1135 is calculated similarly to the kernel size of kernel matrix 1120. Therefore, for sub-feature map 1125A, in the above formula, "i" is 1, "j" is 1, and the acc_ length is 3. Therefore, register 1135 of each accumulator in accumulators 1025A-1025P is initialized with rows (1:3, 1:3) of register group 1130. Since the values ​​in rows 1:3 and columns 1:3 of register group 1130 are all zero during initialization, the values ​​in register 1135 are initialized to have only zero values. In addition to the product values ​​1020A-1020P, register group 1030 also receives the index value corresponding to each data value in sub-feature map 1125A and the index value corresponding to each weight value in kernel matrix 1120, to calculate the portion of register group 1030 that will be loaded into each of the accumulators 1025A-1025P.

[0111] Therefore, at the start of the first iteration, register group 1030 is initialized to have the value 0 shown in register group 1130, and register 1135 of each accumulator in accumulators 1025A-1025P is initialized to have a 3×3 portion from register group 1130. Each accumulator in accumulators 1025A-1025P reads a portion of its corresponding register 1135 in each iteration and adds the current product value (e.g., product value 1020A-1020P) to the product value calculated in the previous iteration. For example, for the first iteration where the weight value corresponding to the index value (1,1) is transferred to multiple multipliers 1005A-1005P, accumulators 1025A-1025P reads portion 1140A of the instance of their corresponding register 1135, as shown... Figure 11 As shown. The order in which parts of register 1135 are read can be determined by the definition of the convolution operation. Therefore, in some embodiments, part 1140A is read first. In other embodiments, according to the definition of the convolution operation, another part of register 1135 can be read first. Accumulators 1025A-1025P add the product values ​​1020A-1020P in the above product matrix to the values ​​in part 1140A. Specifically, accumulators 1025A-1025P can perform the following matrix addition:

[0112]

[0113] In the above equation, the first matrix is ​​the product matrix derived from the product values ​​1020A-1020P, and the second matrix corresponds to the values ​​in portion 1140A. The result of the matrix addition can be stored back into portion 1140A of register 1135 of each accumulator in accumulators 1025A-1025P. Therefore, after the first iteration, register 1135 of each accumulator in accumulators 1025A-1025P has Figure 11 The value shown in register 1145A.

[0114] In the second iteration, the sub-feature map 1125A continues to be loaded into multiple multipliers 1005A-1005P (as discussed above), and the weight value "1" corresponding to index (1, 2) in the kernel matrix 1120 is transferred to each of those multipliers. The product value 1020A-1020P can be represented by combining the product values ​​1020A-1020P in the product matrix z in the second iteration as follows: 0 0 0 5

[0117] The product values ​​1020A-1020P can be transferred to accumulators 1025A-1025P. Accumulators 1025A-1025P can read portion 1140B from register 1145A, which is obtained by shifting one position to the left from the position of portion 1140A. Accumulators 1025A-1025P can perform matrix addition on the values ​​in the product matrix of the second iteration and the values ​​in portion 1140B as follows:

[0118]

[0119] The result of the matrix addition described above can be stored back into part 1140B. Therefore, after the second iteration, register 1135 has the value shown in register 1145B.

[0120] In the third iteration, the sub-feature map 1125A continues to be loaded into multiple multipliers 1005A-1005P (as discussed above), and the weight value "1" corresponding to index (2, 2) in the kernel matrix 1120 is transferred to each of those multipliers. The product value 1020A-1020P can be represented by the product matrix z from the second iteration as: 0 0 0 0

[0123] The product values ​​1020A-1020P can be transferred to accumulators 1025A-1025P. Accumulators 1025A-1025P can read portion 1140C of register 1145B, which is obtained by shifting portion 1140B of register 1145A upwards by one position. Accumulators 1025A-1025P can perform matrix addition on the values ​​in the product matrix of the third iteration and the values ​​in portion 1140C as follows:

[0124]

[0125] The result of the matrix addition described above can be stored in part 1140C. Therefore, after the third iteration, register 1135 has the value shown in register 1145C.

[0126] In the fourth iteration, the sub-feature map 1125A continues to be loaded into multiple multipliers 1005A-1005P (as discussed above), and the weight value "-1" corresponding to index (2, 1) in the kernel matrix 1120 is transferred to each of those multipliers. The product value 1020A-1020P can be represented by the product matrix z from the second iteration as: 0 0

[0128] 0 -5

[0129] The product values ​​1020A-1020P can be transferred to accumulators 1025A-1025P. Accumulators 1025A-1025P can read portion 1140D of register 1145C, which is obtained by shifting the position of portion 1140C one position to the right column by column. Accumulators 1025A-1025P can perform matrix addition on the values ​​in the product matrix of the fourth iteration and the values ​​in portion 1140D as follows:

[0130]

[0131] The result of the matrix addition described above can be stored in part 1140D. Therefore, after the fourth iteration, register 1135 has the value shown in register 1145D.

[0132] Because in Figure 11In the example, there are only four iterations, so after four iterations of the sub-feature map 1125A are completed, accumulators 1025A-1025P load the values ​​stored in register 1135 (e.g., the values ​​in register 1145D) back into register group 1130. Furthermore, the value of register 1145D can be loaded back into register group 1130 at the same location where those values ​​were initially initialized. Therefore, for example, the following formula can be used to determine where in register group 1130 the value from register 1145A will be uploaded:

[0133] Register set = register (i: i + acc_length - 1, j: j + acc_length - 1)

[0134] Therefore, for sub-feature mapping 1125A, the value in register 1145D can be loaded into row (1:3, 1:3) of register group 1130. Thus, after loading the value from register 1145D into register group 1130, register group 1030 can have a value similar to that of register group 1150. Furthermore, the loaded value in register group 1150 can be cyclically shifted one column to the right. For example, the shift operation can be performed according to the following formula:

[0135] Register set = Circular shift(register set, core size - 1, 2)

[0136] The directionality of the shift can depend on the definition of convolution. Therefore, in other embodiments, register group 1150 can be shifted in other ways. Thus, values ​​in register group 1150 can be shifted to obtain the values ​​shown in register 1155. Values ​​in register group 1150 can undergo one or more inline operations in register group 1030. For example, register group 1030 can perform nonlinear corrected linear unit (ReLU) operations and pooling operations. Therefore, in some embodiments, register group 1030 may include a ReLU processing unit and a pooling operation processing unit to perform ReLU and pooling operations, respectively. In other embodiments, separate components may be associated with sparse tensor computation unit 1000 to perform ReLU and pooling operations.

[0137] In some implementations, the ReLU operation may implement the activation function: ƒ(x) = max(0,x). In other implementations, the ReLU operation may implement other activation functions. The ReLU operation may generate one output for each input. Therefore, for A inputs, the ReLU operation may generate A outputs. The pooling operation may then reduce the A results to B results. For example, a pooling operation with a kernel size of 2×2 may reduce the A inputs to A / 4 inputs. Therefore, depending on the kernel size of the pooling operation, register set 1030 may reduce the A inputs to B inputs. Register set 1130 may also be associated with B pooling units, each configured to perform a downsampling operation on one of the A results of the ReLU operation. The pooling operation may perform a max pooling operation to determine the maximum value in a set of cell values, an average pooling operation to determine the average value in a set of cell values, or a sum pooling operation to determine the sum of a set of cell values. In other implementations, other pooling operations may be performed. The result of the pooling operation can represent an output sub-feature map, which can be sent back to the associated sparse tensor memory clusters 400, 500 or sent to another sparse tensor computation unit. In some implementations, the output sub-feature map can be compressed before being sent to the sparse tensor memory clusters 400, 500.

[0138] In some implementations, the output sub-feature map can be combined with other output sub-feature maps generated from other sparse tensor computation units before compression. To combine various output sub-feature maps, in some implementations, the output sub-feature maps can be "stitched" together to obtain the output feature map. For example, in some implementations, if the input feature map is divided into four sub-feature maps [A, B, C, D] (which generate four corresponding output sub-feature maps A', B', C', and D'), then the output feature map can be given as [A', B', C', D'].

[0139] Furthermore, each product value in the product values ​​1020A-1020P is also input into adder 1035 and stored in special accumulator 1040. Adder 1035 can be used to calculate the sum of the grouped product values ​​1020A-1020P. For example, adder 1035 can be used to calculate the sum of a group of P' inputs and P inputs, where P' is less than P. Special accumulator 1040 can handle the accumulation required for fully connected layers and 1×1 convolutions by taking the output of adder 1035 and the result of previous accumulation from register group 1030.

[0140] Now go to Figure 12An exemplary flowchart according to some embodiments of the present disclosure is shown, which outlines the operation of process 1200. Process 1200 may be implemented in sparse tensor computation unit 1000. In some embodiments, a controller may be associated with sparse tensor computation unit 1000 to execute process 1200. The controller may be associated with a memory for storing computer-readable instructions that may be executed by a processor associated with the controller. In other embodiments, scheduling engine 225 may be configured to execute process 1200. Depending on a specific embodiment, process 1200 may include other operations or additional operations.

[0141] After starting at operation 1205, register set 1030 is initialized at operation 1210. As described above, register set 1030 can be initialized to have only zero values ​​at the start of the first iteration. Furthermore, after initializing register set 1030, each of the accumulators 1025A-1025P can be initialized at operation 1215, such that a portion of the register set is loaded into each of those accumulators. This portion of register set 1030 loaded into each of those accumulators is based on the index value of a sub-feature map in the processed input feature map. At operation 1220, data values ​​from the sub-feature map (e.g., sub-feature map 1125A) are input into multiple multipliers 1005A-1005P, as discussed above. In some implementations, operations 1210 / 1215 and 1220 can occur in parallel. At operation 1225, a weight value from kernel matrix 1120 is transferred to each of the multiple multipliers 1005A-1005P.

[0142] At operation 1230, each of the multiple multipliers 1005A-1005P calculates the product between the data values ​​and weight values ​​received in the sub-feature map therein, to obtain a product value (e.g., product values ​​1020A-1020P). The product value can be represented as a product matrix. At operation 1235, the values ​​in the product matrix are added to the values ​​read from the window of that portion loaded into the accumulators 1025A-1025P at operation 1215. For example, in the first iteration, the values ​​in the product matrix can be added to portion 1140A. In the second iteration, the values ​​in the product matrix can be added to portion 1140B, and in the third iteration, the values ​​in the product matrix can be added to portion 1140C, and in the fourth iteration, the values ​​in the product matrix can be added to portion 1140D. Therefore, in each iteration, the window (e.g., portions 1140A-1140D) can be shifted by one position (e.g., from the initial position of portion 1140A—shift left column by column to the position of portion 1140B—shift up row by row to the position of portion 1140C—shift right column by column to the position of portion 1140D). The above shifting pattern can be applied even if register 1135 is larger than 3×3 in size.

[0143] At operation 1240, the result of the addition in each iteration is stored back into accumulators 1025A-1025P. Then, at operation 1245, it is determined whether all iterations have been completed. Again, the number of iterations can depend on the number of non-zero weight values ​​in the weight matrix 1120 and the number of unique weight values ​​that will be transferred to multiple multipliers 1005A-1005P in each iteration. For example, for a 2×2 weight matrix, if a single weight value is transferred in each iteration, and if all four weight values ​​in the weight matrix are non-zero, then process 1200 may include 4 iterations. If additional iterations remain, process 1200 loops back to operation 1210 to continue processing the next iteration. On the other hand, if it is determined at operation 1245 that all iterations have been completed, then at operation 1250, after completing the last iteration, accumulators 1025A-1025P will load the portion obtained at operation 1240 back into register set 1030. At operation 1255, register group 1030 performs a shift operation, and at operation 1260, ReLU and pooling operations are performed on the shifted values ​​in the register group to obtain the output sub-feature map. Optionally, at operation 1265, the output sub-feature map may be compressed. Then, process 1200 terminates at operation 1270 by sending the output sub-feature map to the associated sparse tensor memory clusters 400, 500 or to another sparse tensor computation unit.

[0144] In some implementations of depthwise separable convolution (e.g., where the feature map has multiple channels), the result from each channel can be stored as an intermediate output sub-feature map. For example, operations 1205 to 1255 can be performed on the sub-feature map for each channel, and the result of those operations can be stored as an intermediate output sub-feature map. In some implementations, ReLU and pooling operations may not be performed on the intermediate output sub-feature maps. Thus, depending on the number of channels, multiple intermediate output sub-feature maps can be obtained. For example, for three channels, three intermediate output sub-feature maps can be obtained (e.g., depthwise convolution or spatial convolution performed independently on each channel of the input). Each intermediate output sub-feature map in the intermediate output sub-feature maps can then be combined by applying a 1×1 filter according to the given hyperparameters of the neural network (e.g., projecting the channels output by the depthwise convolution onto a pointwise convolution in the new channel space, such as a 1×1 convolution). ReLU and pooling operations of operation 1260 can then be performed on the combined intermediate output sub-feature maps to obtain the output sub-feature map.

[0145] Now for reference Figure 13 An exemplary block diagram of a sparse tensor computation unit 1300 according to some embodiments of the present disclosure is shown. The sparse tensor computation unit 1300 is similar to one of a plurality of sparse tensor computation units 805A-805M or 905A-905M. The sparse tensor computation unit 1300 is configured to perform various machine learning operations (such as multiplication, addition, etc.) that may be performed during convolution operations in a CNN. The sparse tensor computation unit 1300, like the sparse tensor computation unit 1000, includes a plurality of multipliers 1305A-1305P. Each of the plurality of multipliers 1305A-1305P is configured similarly to the plurality of multipliers 1005A-1005P and is therefore not described further.

[0146] Furthermore, each of the multiple multipliers 1305A-1305P is configured to receive data values ​​(e.g., data values ​​1310A-1310P) from a sub-feature map (e.g., sub-feature map 1125A) and weight values ​​(e.g., weight values ​​1315A-1315P) to be applied to the sub-feature map from a kernel matrix (e.g., kernel matrix 1120). Each of the multiple multipliers 1305A-1305P computes the product between the data values ​​(e.g., data values ​​1310A-1310P) and the weight values ​​stored therein (e.g., weight values ​​1315A-1315P) to generate product values ​​1320A-1320P, respectively. Similarly, similar to the sparse tensor computation unit 1000, the sparse tensor computation unit 1300 also processes the sub-feature map 1125A in multiple iterations (e.g., four iterations, as discussed above). In each iteration, different weight values ​​from kernel matrix 1120 can be applied to sub-feature map 1125A.

[0147] In some implementations, each of the multiple multipliers 1305A-1305P may also receive the index value of the data values ​​1310A-1310P and the index value of each weight value among the weight values ​​1315A-1315P, respectively. For example, multiplier 1305A may receive the index value of the data value 1310A and the index value of each weight value from the weight values ​​of the kernel matrix 1120. Based on the index value, each of the multiple multipliers 1305A-1305P may determine the index value of the position where the product value calculated by that multiplier (e.g., product values ​​1320A-1320P) will be located in the output sub-feature map. In some implementations, each of the multiple multipliers 1305A-1305P may use the following formula to calculate the index value of the product value 1320A-1320P:

[0148] XW_row_idx = X_row_idx + ((W_ROW_LEN – 1) - W_row_idx)

[0149] XW_col_idx = X_col_idx + ((W_COL_LEN – 1) - W_col_idx)

[0150] In the above formulas, indices start from 0, and XW_row_idx and XW_col_idx are the row and column indices of the product values ​​(e.g., product values ​​1320A-1320P) in the output sub-feature map, respectively. X_row_idx and X_col_idx are the row and column indices of the first multiplicand (e.g., data values ​​1310A-1310P), respectively. W_row_idx and W_col_idx are the row and column indices of the second multiplicand (e.g., weight values ​​1315A-1315P), respectively. W_ROW_LEN and W_COL_LEN are the dimensions of the kernel matrix 1120 (e.g., the kernel size). Furthermore, XW-row_idx can be the same as the X_row_idx of the offset filter row length, and the filter coefficient row indices use values ​​from 0 to W_ROW_LEN-1.

[0151] After calculating the index values ​​of the product values ​​1320A-1320P, each of the multiple multipliers 1305A-1305P can transfer its corresponding product value and calculated index value to the accumulator 1325. Although the multiple multipliers 1305A-1305P have been described as calculating the index values ​​of the product values ​​1320A-1320P, in some embodiments, the accumulator 1325 may alternatively receive various index values ​​of the first and second multiplicands to calculate the index values ​​of those product values. In other embodiments, another component of the accelerator 200 can calculate the index values ​​of the product values ​​1320A-1320P in the output sub-feature map.

[0152] Therefore, each of the product values ​​1320A-1320P and the calculated index value is input into accumulator 1325. In some embodiments, accumulator 1325 may be a combination of "P" accumulators (where "P" is the total number of multiple multipliers 1305A-1305P), and each accumulator may resemble multiple accumulators 1025A-1025P. Therefore, accumulator 1325 is configured to resemble accumulators 1025A-1025P. Similar to accumulators 1025A-1025P, accumulator 1325 may be configured to add each product value in product values ​​1320A-1320P to the sum calculated in the last iteration. For example, accumulator 1325 may add the product value 1320A of the current iteration to the product value 1320A from a previous iteration. Therefore, accumulator 1325 may perform the following operations:

[0153] a1 = a1 + z1

[0154] a² = a² + z²

[0155] …

[0156] aP = aP + zP

[0157] In the above equations, z1 is the product value 1320A, z2 is the product value 1320B, zP is the product value 1320P, and so on. At the start of the first iteration, each of the values ​​a1, a2, ..., aP can be initialized to zero. The results 1330A-1330P (including the summation results (e.g., a1, a2, ..., aP)) and the index values ​​calculated for each of these results can be sent from accumulator 1325 to multilevel interconnection network (“MIN”) 1335. Result 1330A may include the summation result a1, and the index value calculated by multiplier 1305A for the product value 1320A. Similarly, result 1330B may include the summation result a2, and the index value calculated by multiplier 1305B for the product value 1320B, and so on.

[0158] MIN 1335 can be used for local interleaving and connecting accumulator 1325 to variable accumulator register (“VAR”) 1340. Additional details of MIN 1335 can be found in U.S. Application No. 15 / 627,042, filed June 19, 2017, the entire contents of which are incorporated herein by reference. MIN 1335 can include various control states. In some embodiments, the total number of inputs to the read MIN (e.g., MIN 1355 described below) is (2k-1)^2. Since P values ​​out of these (2k-1)^2 values ​​need to be retrieved, read MIN (e.g., MIN 1355) can be used. The total number of locations to which write MIN (e.g., MIN 1335) is connected is (2k-1)^2. Since P values ​​out of these (2k-1)^2 values ​​need to be written, write MIN (e.g., MIN 1335) can be used. In some implementations, if both of the product values ​​1320A-1320P have the same calculated index value, then MIN 1335 can classify the results 1330A-1330P. Therefore, writing to MIN (e.g., MIN 1335) can be used to write to VAR 1340, and reading from MIN (e.g., MIN 1355) can be used to read from VAR 1340. Thus, MIN 1335 can transfer the classification results 1345A-1345P (including the index values ​​calculated by multiple multipliers 1305A-1305P) to VAR 1340.

[0159] Therefore, the MIN 1335 and MIN 1355 can provide bidirectional connectivity (e.g., one direction for reading and the other for writing) on ​​the first P-port of the Benes network (or, in the case of no sparsity in the input sub-feature map) (the total number of ports can be (2k-1)^2), where the number of P-ports can be based on the percentage of non-zero values ​​in the input sub-feature map. For example, if there are 60% non-zero values ​​in the input sub-feature map and the kernel matrix is ​​3×3, then 6 P-ports can be chosen. In some implementations, the P-ports can be chosen independently of the average sparsity and can be based on various hardware considerations.

[0160] VAR 1340 is initialized with a portion from accumulator buffer 1350, similar to accumulators 1025A-1025P. Accumulator buffer 1350 is similar to register set 1030. However, accumulator buffer 1350 does not perform shift operations. Similar to register set 1030, accumulator buffer 1350 can have the size of an output sub-feature map, as discussed above relative to the register set. For example, using... Figure 11 For example, the accumulator buffer 1350 can have a size of 7×7. Furthermore, at the start of the first iteration, the accumulator buffer 1350 can be initialized to have only zero values, as discussed above.

[0161] Additionally, at the start of the first iteration, a portion of the accumulator buffer 1350 can be loaded into VAR 1340. The size of VAR 1340 can be calculated using the following formula: (2k-1) × (2k-1), where k is the kernel size of kernel matrix 1120. Since in Figure 11 In the example, the kernel size k of kernel matrix 1120 is 2, therefore the size of VAR 1340 is 3×3. Furthermore, the 3×3 portion copied into VAR 1340 from accumulator buffer 1350 can be determined based on the following formula:

[0162] VAR = Accumulator Buffer (i: i + acc_length - 1, j: j + acc_length - 1)

[0163] In the above formula, the first term corresponds to the row number (e.g., row index value) of the accumulator buffer 1350, and the second term corresponds to the column number (e.g., column index value) of the accumulator buffer. Furthermore, in the above formula, "i" is the start of the row number, and "j" is the start of the column number in the sub-feature map 1125A within the filled feature map 1115, and the acc_ length is the size of the accumulator buffer 1350. For example, since the size of the accumulator buffer 1350 in the above example is 3×3, the acc_ length is 3. Therefore, for example, to process the sub-feature map 1125A, in the above formula, "i" is 1, "j" is 1, and the acc_ length is 3. Therefore, at the start of the first iteration, rows 1:3 and columns 1:3 of the accumulator buffer 1350 can be loaded into VAR 1340.

[0164] After receiving the classification results 1345A-1345P and the calculated index values ​​from MIN 1335, VAR 1340 stores the results in the appropriate index values ​​of that portion, copied from accumulator buffer 1350. After storing the results 1345A-1345P in the appropriate index values ​​in VAR 1340, VAR can transfer the classification results 1345A-1345P to MIN 1355, which is constructed in a manner similar to MIN 1335. MIN 1355 can then send the classification results 1345A-1345P back to accumulator 1325 for use in the next iteration.

[0165] Furthermore, after completing all iterations of processing the sub-feature map 1125A, VAR 1340 can send the results stored therein back to accumulator buffer 1350. For example, if rows 1:3 and columns 1:3 are copied from accumulator buffer 1350 to VAR 1340 at the start of the first iteration, then at the end of the last iteration, the results from VAR are loaded back into rows 1:3 and 1:3 of the accumulator buffer. Therefore, the portion of accumulator buffer 1350 that was loaded into VAR 1340 at the start of the first iteration is replaced by the results from VAR 1340 at the end of the last iteration.

[0166] In addition to sending the product values ​​1320A-1320P to accumulator 1325, these product values ​​are also input into adder 1360. The result from adder 1360 is stored in special accumulator 1365. Adder 1360 and special accumulator 1365 are similar to adder 1035 and special accumulator 1040, respectively.

[0167] As discussed above, ReLU and pooling operations 1370 are performed on the values ​​stored in accumulator buffer 1350 after the last iteration to obtain the output sub-feature map. The results of ReLU and pooling operations 1370 may optionally be compressed in compression block 1375, as discussed above. The compressed results may be sent back to DRAM 215 via the associated sparse tensor memory cluster or sent to another sparse tensor computation unit.

[0168] Now for reference Figure 14 An example of a sparse tensor computation unit 1400 according to some embodiments of the present disclosure is shown. The sparse tensor computation unit 1400 is substantially similar to the sparse tensor computation unit 1300. For example, similar to the sparse tensor computation unit 1300, the sparse tensor computation unit 1400 includes a plurality of multipliers 1405A-1405P that receive data values ​​1410A-1410P and weight values ​​1415A-1415P to generate product values ​​1420A-1420P. Each of the plurality of multipliers 1405A-1405P may also receive index values ​​of the data values ​​1410A-1410P and weight values ​​1415A-1415P to compute the index values ​​of the product values ​​1420A-1420P, as explained above.

[0169] The sparse tensor computation unit 1300 is configured to receive one unique weight value at a time. In other words, in each iteration, the same weight value is transmitted to each of the multiple multipliers 1305A-1305P in the sparse tensor computation unit 1300. However, the sparse tensor computation unit 1400 is configured to process more than one unique weight value in each iteration. In some embodiments, it may be necessary to process more than one unique weight value at a time. In this case, multiple weight values ​​can be broadcast to multiple multipliers 1405A-1405P at once. For example, a first group of the multiple multipliers 1405A-1405P may receive a first weight value, a second group of the multiple multipliers may receive a second weight value, and so on. For example, when using two weight values ​​in each iteration, the first weight value can be transferred to Q multipliers out of multiple multipliers 1405A-1405P (where Q is the number of non-zero values ​​in a particular sub-feature map), and the second weight value can be transferred to (PQ) multipliers out of those multiple multipliers. The number of unique weight values ​​that can be used in each iteration can be calculated using the following formula: max(1, ceiling(PQ)), where Q is the number of non-zero values ​​in a particular sub-feature map, and P is the number of multiple multipliers 1405A-1405P involved.

[0170] The product values ​​1420A-1420P and the calculated index value can be transferred to MIN 1425. Specifically, MIN 1425 can be a write arbitrator MIN with "P" FIFO buffers and P×P control states. If more than one product value among the product values ​​1420A-1420P in the output sub-feature map has the same calculated index value, MIN 1425 can send one of the product values ​​for that index value while storing the remaining product values ​​in the buffer. MIN 1425 then sends the product values ​​1420A-1420P and the calculated index value to accumulator 1430. Accumulator 1430 is similar to accumulator 1425. Furthermore, MIN 1435, VAR 1440, MIN 1445, adder 1450, special accumulator 1460, accumulator buffer 1465, ReLU and pooling operations 1470, and compression block 1475 are configured similarly to MIN 1335, VAR 1340, MIN 1355, adder 1360, special accumulator 1365, accumulator buffer 1350, ReLU and pooling operations 1370, and compression block 1375, and therefore will not be described further for simplicity. Each of these components processes as described above. Figure 14 The sub-feature mapping diagram 1125A described in the document.

[0171] Now go to Figure 15 An example of a sparse tensor computation unit 1500 according to some embodiments of the present disclosure is shown. The sparse tensor computation unit 1500 is substantially similar to the sparse tensor computation unit 1400. For example, similar to the sparse tensor computation unit 1400, the sparse tensor computation unit 1500 includes a plurality of multipliers 1505A-1505P that receive data values ​​1510A-1510P and weight values ​​1515A-1515P to generate product values ​​1520A-1520P and compute index values ​​for those product values. Also similar to the sparse tensor computation unit 1400, the sparse tensor computation unit 1500 is configured to receive a plurality of weight values ​​in each iteration. The product values ​​1520A-1520P and the computed index values ​​can be transferred to MIN 1525, which can be constructed in a manner similar to MIN 1425. MIN 1525 transfers values ​​1530A-1530P to adder block 1535. Adder block 1535 can perform the following addition operations:

[0172] S1 = b1 + b2

[0173] S2 = b2 + b3

[0174] S3 = b3 + b4

[0175] …

[0176] SP = bP + b1

[0177] The summed values ​​(S1, S2, ..., SP) can be considered speculative calculations. If two weight values ​​are sent to the sparse tensor computation unit 1500, it may be necessary to add the two product values ​​generated by these two weight values ​​together before accumulation. The summed values ​​(S1, S2, ..., SP) can be transmitted as values ​​1540A-1540P to the selection unit 1545. The selection unit 1545 can be a set of multiplexers configured to perform the following operations:

[0178] R1 = MUX(b1, SP, S1)

[0179] R2 = MUX(b2, S1, S2)

[0180] R3 = MUX(b3, S2, S3)

[0181] …

[0182] RP = MUX(bP, S(P-1),SP)

[0183] In some implementations, the selection of the multiplexer can be controlled based on the output index value of each product value. If two adjacent product values ​​share the same output index value, the sum of the adjacent product values ​​can be selected.

[0184] Although the selection unit 1545 has been described as having a set of multiplexers, in other embodiments, other elements configured to select one input from multiple inputs based on selection criteria may be used.

[0185] The results 1550A-1550P (e.g., R1, R2, ..., RP) can be transferred to accumulator 1555. Accumulator 1555 is similar to accumulator 1430 and is configured to perform the following addition operations:

[0186] a1 = R1 + a2

[0187] a2 = R2 + a3

[0188] a3 = R3 + a4

[0189] …

[0190] aP = RP + a1

[0191] The result from accumulator 1555 can be sent to MIN 1560, which is similar to MIN 1435. Furthermore, the result from accumulator 1555 can be processed in VAR 1565, accumulator buffer 1570, MIN 1575, adder 1580, special accumulator 1585, ReLU and pooling operation 1590, and compaction block 1595, similar to the processing performed in VAR 1435, accumulator buffer 1465, MIN 1445, adder 1450, special accumulator 1460, ReLU and pooling operation 1470, and compaction block 1475, and therefore will not be explained again.

[0192] Go to Figure 16 An exemplary flowchart according to some embodiments of the present disclosure is shown, which outlines the operation of process 1600. Depending on the embodiment, process 1600 may include other operations or additional operations. Process 1600 may be implemented in sparse tensor computation unit 1300, sparse tensor computation unit 1400, or sparse tensor computation unit 1500. In some embodiments, the sparse tensor computation unit implementing process 1600 may include a controller executing process 1600. The controller may be associated with a memory for storing computer-readable instructions that may be executed by a processor associated with the controller. In other embodiments, scheduling engine 225 may be configured to execute process 1600. Process 1600 may be implemented during standard convolution operations in a CNN. Process 1600 may also be implemented in a fully connected 1×1 convolution operation in which a 1×1 weight matrix is ​​applied.

[0193] After starting at operation 1605, the accumulator buffers (e.g., accumulator buffer 1350, accumulator buffer 1465, accumulator buffer 1570) of the sparse tensor computation units (e.g., sparse tensor computation unit 1300, sparse tensor computation unit 1400, or sparse tensor computation unit 1500) of implementation process 1600 are initialized at operation 1610. Operation 1610 is similar to operation 1210. As described above, at the start of the first iteration, the accumulator buffers can be initialized to have only zero values. Furthermore, after initializing the accumulator buffers at operation 1610, the VARs (e.g., VAR 1340, VAR 1440, VAR 1565) of the sparse tensor computation units of implementation process 1600 are initialized at operation 1615, such that a portion of the accumulator buffers is loaded into the VARs.

[0194] At operation 1620, data values ​​from the sub-feature map (e.g., sub-feature map 1125A) are input into multiple multipliers (e.g., multiple multipliers 1305A-1305P, multiple multipliers 1405A-1405P, multiple multipliers 1505A-1505P) of the sparse tensor computation unit of implementation process 1600, as discussed above. At operation 1620, index values ​​corresponding to the data values ​​may also be input into these multiple multipliers. At operation 1625, one or more weight values ​​(e.g., from kernel matrix 1120) are transferred to each of the multiple multipliers (e.g., multiple multipliers 1305A-1305P, multiple multipliers 1405A-1405P, multiple multipliers 1505A-1505P) of the sparse tensor computation unit of implementation process 1600, as discussed above. The index values ​​of these weights can also be input into those multipliers. The number of weight values ​​transferred to each of the multiple multipliers can depend on the type of convolution being performed. For example, in some implementations, a standard convolution operation can transfer two unique weight values ​​to each of the multiple multipliers, while a fully connected 1×1 convolution operation can transfer a single weight value to each of those multipliers. In some implementations, operations 1610 / 1615 and 1620 / 1625 can occur in parallel.

[0195] At operation 1630, each of the plurality of multipliers (e.g., multipliers 1305A-1305P, multipliers 1405A-1405P, and multipliers 1505A-1505P) computes the product between the data values ​​and weight values ​​received in the sub-feature map therein, to obtain product values ​​(e.g., product values ​​1320A-1320P, product values ​​1420A-1420P, and product values ​​1520A-1520P). The product values ​​can be represented as a product matrix. At operation 1635, the plurality of multipliers (e.g., multipliers 1305A-1305P, multipliers 1405A-1405P, and multipliers 1505A-1505P) also compute the index values ​​of those product values, as discussed above.

[0196] At operation 1640, if two product values ​​share the same computation index, the MIN (e.g., MIN 1335, MIN 1435, MIN 1460) of the sparse tensor computation unit implementing process 1600 classifies the product values ​​such that product values ​​sharing the same index are adjacent to each other in the VAR (e.g., VAR 1340, VAR 1440, VAR 1565) of the sparse tensor computation unit implementing process 1600. At operation 1645, the classification result from the MIN is stored in the VAR. Then, at operation 1650, it is determined whether all iterations have been completed. If there are additional iterations remaining, process 1600 loops back to operation 1610 to continue processing the next iteration. On the other hand, if it is determined at operation 1650 that all iterations have been completed, then at operation 1655, the values ​​in the VARs obtained at operation 1645 after the completion of the last iteration (e.g., VAR 1340, VAR 1440, VAR 1565) are loaded back into the accumulator buffers (e.g., accumulator buffer 1350, accumulator buffer 1465, accumulator buffer 1570). At operation 1660, ReLU and pooling operations are performed on the values ​​in the accumulator buffers to obtain the output sub-feature map. Optionally, at operation 1665, the output sub-feature map may be compressed. Then, process 1600 terminates at operation 1670 by sending the output sub-feature map to the associated sparse tensor memory clusters 400, 500 or to another sparse tensor computation unit.

[0197] In some implementations, the output sub-feature map can be further subdivided into additional sub-feature maps based on sparsity (as described above). Figure 3A and Figure 3B (As discussed in the previous section), to balance the number of zero and non-zero values ​​in each of the other sub-feature maps, and to balance the workload in each sparse tensor computation unit in the sparse tensor computation unit of the sparse tensor computation cluster 230. A similar rebalancing of the output sub-feature maps can be performed in process 1200.

[0198] In some implementations of depthwise separable convolution (e.g., where the feature map has multiple channels), the result from each channel can be stored as an intermediate output sub-feature map. For example, operations 1605 to 1665 can be performed on the sub-feature map for each channel, and the result of those operations can be stored as an intermediate output sub-feature map. In some implementations, ReLU and pooling operations may not be performed on the intermediate output sub-feature maps. Therefore, multiple intermediate output sub-feature maps can be obtained depending on the number of channels. For example, for three channels, three intermediate output sub-feature maps can be obtained. Each intermediate output sub-feature map in the intermediate output sub-feature map can then be combined by applying a 1×1 filter. Then, ReLU and pooling operations of operation 1660 are performed on the combined intermediate output sub-feature maps to obtain the output sub-feature map.

[0199] Now go to Figure 17 An exemplary flowchart according to some embodiments of the present disclosure is shown, which outlines the operation of process 1700. Depending on the specific embodiment, process 1700 may include additional operations or other operations. Process 1700 may be implemented by accelerator 200. In some embodiments, process 1700 may be implemented by scheduling engine 225 of accelerator 200. In other embodiments, process 1700 may be implemented by another component of accelerator 200. Process 1700 begins at operation 1705, where one or more machine learning operations will be performed on new input data (e.g., input image 205) in a machine learning application. For example, process 1700 may be implemented when image classification is to be completed to identify one or more features in the input image. Thus, at operation 1710, the input image or a feature map associated with the input image may be input into accelerator 200.

[0200] The input image can be represented by a pixel array based on the size, height, and color scheme of the input image. Simply as an example and not intended to be limiting in any way, process 1700 is explained below relative to an input image of size 1024 × 1024 × 3. Therefore, the input image is 1024 pixels wide and 1024 pixels high, and has 3 color channels. Thus, the input image can be viewed as a combination of three feature maps, one for each color channel, and each feature map having a size of 1024 × 1024 × 1. In some embodiments, the input image can be input into accelerator 200, and the accelerator can derive the feature map from the input image. In other embodiments, the conversion of the input image into a feature map can occur outside of accelerator 200 (performed by another component on a host device (e.g., host device 105) associated with accelerator 200), and the feature map, instead of the input image, can be input into the accelerator. Furthermore, the input image or its feature map (whichever is input into the accelerator 200) can be stored in the accelerator's DRAM 215.

[0201] At operation 1715, accelerator 200 receives a weight matrix (e.g., weight matrix 220). Although operation 1715 has been described as occurring after operation 1710 (in which the input image or its feature map is received by accelerator 200), in some embodiments, operation 1715 may occur before operation 1710. In some embodiments, operations 1710 and 1715 may occur simultaneously or substantially simultaneously.

[0202] In some implementations, accelerator 200 can be configured for a neural network with multiple layers. For example, in some implementations, accelerator 200 can be configured for a convolutional neural network with multiple convolutional layers. In some implementations, each of the multiple convolutional layers can have a specific weight matrix that can be applied to the feature map of that layer. Therefore, in such implementations, accelerator 200 can receive multiple weight matrices, one of which is configured for a layer. Each weight matrix can include weight values ​​that can be applied to the feature map. Specifically, multiplication, addition, and other operations can be performed between the corresponding weight values ​​and the data values ​​of the input feature map. In some implementations, the weight matrix can also be compressed. In some implementations, the weight matrix can be stored at least temporarily in DRAM 215.

[0203] In some implementations, the weight matrix may be reordered before or after compression, and static sparsity analysis may be performed on the weight matrix. Static sparsity analysis can be performed before or after compressing the weight matrix. In some implementations, static sparsity analysis may be performed by accelerator 200, while in other implementations, it may be performed outside the accelerator. Static sparsity analysis identifies weight values ​​that are zero in the weight matrix. If a particular weight value is zero, then multiplication with that zero weight value is also zero. Therefore, the results of those multiplication operations can be directly encoded as zero, rather than allocating resources to multiply multiple zero weight values ​​with values ​​in the feature map. Furthermore, since the weight values ​​that are zero in the weight matrix are known in advance, and these weight values ​​do not change from one feature map to another, the sparsity in the weight matrix is ​​static.

[0204] Therefore, the weight matrices can be compressed and reordered, and static sparsity analysis can be performed to identify zero weight values ​​and obtain input weight tensors from each weight matrix. Each input weight value can be stored in DRAM 215 and is assigned for computation with the input feature map based on a load and store schedule stored in accelerator 200.

[0205] At operation 1720, each input feature map of the input image is compressed, as discussed above. Although operation 1720 is described after operation 1715, in some embodiments, operation 1720 can occur at any time after the feature maps of the input image are available. Each feature map in the feature map can be compressed by recursively dividing each feature map in the feature map into smaller cell-size portions until a desired compression criterion is reached. The desired compression criterion can be based on the number of compression levels or a threshold minimum cell size, as discussed in more detail in U.S. Application No. 16 / 726,084 mentioned above. The compression operations discussed throughout this disclosure can be performed as discussed in U.S. Application No. 16 / 726,084.

[0206] At operation 1725, each input feature map of the input image is divided into multiple sub-feature maps, as described above. Figure 3A and Figure 3BAs discussed in [the previous section], by dividing the input feature map into multiple sub-feature maps, each of these sub-feature maps can be processed in parallel. Furthermore, the input feature map can be divided into multiple sub-feature maps such that each of these sub-feature maps has the same or substantially similar sparsity. In some implementations, the sparsity of each of these sub-feature maps satisfies a predetermined threshold (e.g., the predetermined percentage difference threshold discussed above). Therefore, each of these sub-feature maps can have the same or similar number of zero values ​​and / or non-zero values. Moreover, by dividing the input feature map into multiple sub-feature maps, the dynamic sparsity of the input feature map can be transformed into static sparsity or close to the static sparsity of each of these sub-feature maps.

[0207] At operation 1730, each sub-feature map in the plurality of sub-feature maps can be assigned to a sparse tensor computation unit based on the proximity of each sub-feature map in the current storage location of the computation unit (e.g., sparse tensor computation units 1000, 1300, 1400, 1500). At operation 1735, each sub-feature map in the plurality of sub-feature maps is stored in a sparse tensor feature map memory unit, which is associated with a sparse tensor computation unit for processing a specific sub-feature map in the plurality of sub-feature maps. In some operations, if a specific sub-feature map is larger than the storage capacity of the sparse tensor feature map memory unit in which the sub-feature map will be stored, a time-folding operation can be performed. In some embodiments, the time-folding operation can be considered similar to the above-described... Figure 3A and Figure 3B The sub-feature partitioning described in [the document] is based on the feature map size during the temporal folding operation. For example, if the maximum supported input image size is 1024×1024, and if there is an input image of size 2048×2048, the large input image can be partitioned into four input feature maps, each with a size of 1024×1024.

[0208] At operation 1735, the weight values ​​of a specific sub-feature map to be applied to the plurality of sub-feature maps are also transferred to the sparse tensor weight memory associated with the sparse tensor computation unit. When the sparse tensor computation unit becomes available, the sparse tensor feature map memory unit can transfer the specific sub-feature map to that sparse tensor computation unit. The sparse tensor weight memory can also transfer the weight matrix stored therein to the sparse tensor computation unit.

[0209] At operation 1740 and as mentioned above, relative to... Figures 10 to 16 The sparse tensor computation unit discussed here processes the received sub-feature map to generate an output sub-feature map. Each sparse tensor computation unit can send its output sub-feature map back to DRAM 215 or to another sparse tensor computation unit. In some embodiments, it can be based on... Figure 3A and Figure 3B One or more output sub-feature maps are further subdivided and processed in a sparse tensor computation unit. At operation 1745, once all output sub-feature maps have been processed, they can be combined to generate the output feature map. Process 1700 ends at operation 1750.

[0210] In some implementations, the number of non-zero values ​​in the output sub-feature map can be measured at the output of each sparse tensor computation unit. When generating the output sub-feature map, it might be possible to redirect the output sub-feature map of one sparse tensor computation unit to another to balance sparsity. However, in some implementations, it might be easier and more efficient to balance the sparsity in the output sub-feature map by generating all sub-feature maps and storing them in a sparse tensor feature map memory, while simultaneously feeding them to the sparse tensor computation unit of the next convolutional layer. To balance sparsity in this way, another network can be used to provide the connection between the sparse tensor feature map memory and the sparse tensor computation unit. When processing depthwise separable convolutions, multiple intermediate feature maps can be used to perform pointwise or 1×1 convolutions. These intermediate feature maps may not be sparse because they only undergo convolution and not ReLU (activation). It may also be necessary to store the output of each intermediate feature map back to DRAM. Sparsity occurs at the final output of pointwise or 1×1 convolutions because these convolutions include ReLU (activation).

[0211] Now go to Figures 18A to 18D An example of an input feature map 1800 in a sparse tensor computation unit 1300 according to some embodiments of the present disclosure is shown. It should be understood that the input feature map 1800 is merely an example and is not intended to be limiting in any way. The present disclosure can be used to process input feature maps of any size and containing any data values. The input feature map 1800 can be generated from an input image 205. Specifically, as... Figure 18A As shown, the input feature map 1800 may include multiple cells, each cell being a row (which extends in the X direction 1105; see also...) Figure 11 ) and column (extending in the Y direction 1110; see Figure 11It is formed at the intersection of the input feature map 1800. Each cell in the multiple cells of the input feature map 1800 includes the data value that will be processed in the sparse tensor computation unit 1300.

[0212] Despite Figure 18A Not shown in the diagram, but in some embodiments, similar to input feature map 1100, input feature map 1800 may be filled with zero values ​​on all sides. Furthermore, in some embodiments, sub-feature maps may be created from input feature map 1800 (or from a filled feature map when padding is used). In some embodiments, input feature map 1800 may be segmented into sub-feature maps within partition blocks 210 or other components of accelerator 200, as described above. Figure 3A and Figure 3B As discussed in [the document]. For example, and as [the document discusses]. Figure 18A As shown, the input feature map 1800 can be divided into multiple sub-feature maps 1805A-1805P. Each of the multiple sub-feature maps 1805A-1805P can be processed sequentially in the sparse tensor computation unit 1300 (or in parallel in multiple instances of the sparse tensor computation unit 1300). In some embodiments, the multiple sub-feature maps 1805A-1805P can be processed in the order shown by arrow 1810. For example, sub-feature map 1805A can be processed first, followed by sub-feature maps 1805B-1805D. Next, sub-feature maps 1805E-1805H can be processed, followed by sub-feature maps 1805I-1805L, and so on. Furthermore, within each of the multiple sub-feature maps 1805A-1805P, data values ​​can be processed in the order shown by arrow 1815. In other embodiments, the order in which the multiple sub-feature maps 1805A-1805P are processed and / or the order in which the data values ​​of each sub-feature map in the multiple sub-feature maps are processed can vary. Additionally, in some embodiments, only non-zero data values ​​can be input into the sparse tensor computation unit 1300.

[0213] Although each of the multiple sub-feature maps 1805A-1805P comprises two rows and two columns and all of those sub-feature maps have the same size, it should be understood that in other embodiments, those sub-feature maps may have different sizes, may include different numbers of rows and columns, and the data values ​​within each of the multiple sub-feature maps 1805A-1805P may vary. Furthermore, depending on the number of multipliers 1305A-1305P in the sparse tensor computation unit 1300, multiple data values ​​of each of the multiple sub-feature maps 1805A-1805P can be processed in parallel. For illustrative purposes, the explanation is relative to sub-feature map 1805A. Figures 18B to 18D However, other sub-feature maps in multiple sub-feature maps 1805B-1805P can be handled in a similar manner. Furthermore, the description assumes that the number of multiple multipliers 1305A-1305P is three. Figures 18A to 18D Since sub-feature map 1805A includes four non-zero data values, each iteration can include two rounds. In the first round, three data values ​​of sub-feature map 1805A can be processed, and in the second round, the last data value of the sub-feature map can be processed. Since the data values ​​are processed in the order shown by arrow 1815, in the first round, the data values ​​with indices (0, 0), (0, 1), and (1, 0) can be processed, while in the second round, the data value with index (1, 1) can be processed.

[0214] The data values ​​of sub-feature map 1805A can be combined with the weight values ​​of kernel matrix 1820. It should be understood that kernel matrix 1820 is merely an example and is not intended to be limiting in any way. Kernel matrix 1820 may assume other dimensions (e.g., the number of rows and columns may differ from those shown), and the values ​​within the kernel matrix may also vary. Furthermore, although each of the multiple sub-feature maps 1805A-1805P has the same number of rows and columns as kernel matrix 1820, in some embodiments, one or more of the multiple sub-feature maps may have a different number of rows and / or columns than the kernel matrix. The weight values ​​in kernel matrix 1820 can be applied in the specified order as indicated by arrow 1825. Moreover, since sparse tensor computation unit 1300 is configured to use a single weight value in each iteration, a single weight value is broadcast to each of the multiple multipliers 1305A-1305P in each iteration.

[0215] Therefore, to process the sub-feature map 1805A, data values ​​from this sub-feature map can be input into multiple multipliers 1305A-1305P. Assume there are three multipliers and... Figure 18BAs shown, the first three data values ​​can be input into multiple multipliers 1305A-1305P. Specifically, the data value "3" can be input into multiplier 1305A, the data value "9" can be input into multiplier 1305B, and the data value "6" can be input into multiplier 1305C. Furthermore, each multiplier in multipliers 1305A-1305C can receive a first weight value "-6" (e.g., ...). Figure 18B As shown in the diagram, the multiplier calculates the product between its corresponding data value and the first weight value. Each multiplier in multipliers 1305A-1305C can also receive the index value of the data value received by each of those multipliers, as well as the index value of the first weight value. Based on the received index value, each multiplier in multipliers 1305A-1305C can calculate the result index value of the location in the output sub-feature map where the result of the multiplication will be stored. In some implementations, multipliers 1305A-1305C can use the method described above. Figure 13 The formula discussed in the document is used to calculate the resulting index value.

[0216] The product value and result index value from multipliers 1305A-1305C can be sent to accumulator 1325. Accumulator 1325 can also receive addition results from previous iterations from MIN 1355, such as... Figure 18B As shown. Specifically, MIN 1355 can also receive result index values ​​and read the values ​​corresponding to these result index values ​​from VAR 1340, and send those values ​​to accumulator 1325, as shown. Figure 18B As shown. Then, accumulator 1325 can add the product value of the current iteration to the product value of the previous iteration. Accumulator 1325 can transfer the result of the addition and the result index value to MIN 1335. MIN 1335 can write the result of the addition to VAR 1340 at the index position identified by the result index value. As described above, VAR 1340 is initialized with a portion from accumulator buffer 1350. Therefore, after writing the result of the addition to VAR 1340 at the index position identified by the result index value, VAR 1340 can have a value as shown in output mapping diagram 1830. Figure 18B Only a portion of the output map is shown.

[0217] In the second round of the first iteration, the remaining data value "-5" of the sub-feature map 1805A is input into multiplier 1305A. The first weight value "-6" is also input into multiplier 1305A, as follows: Figure 18CAs shown. The index values ​​of the data values ​​and the index values ​​of the first weight values ​​are also input into multiplier 1305A. The product value and the result index value are sent to accumulator 1325, which adds the product value to the product value from the previous iteration, and the result is sent to VAR 1340 via MIN 1335, as shown. Figure 18C The output mapping is shown in Figure 1830.

[0218] Now for reference Figure 18D In the first round of the second iteration, the data values ​​"3", "9", and "6" are input into multipliers 1305A, 1305B, and 1305C, respectively. Additionally, the second weight value "6" is input into multipliers 1305A-1305C, as follows: Figure 18D As shown. Additionally, the index values ​​of the data values ​​and the second weight values ​​can be input into multipliers 1305A-1305C. The data values ​​and the second weight values ​​can be combined in the sparse tensor computation unit 1300, as discussed above relative to the first round of the first iteration. The output of the first round of the second iteration can be stored in the output map graph 1830, as shown. Figure 18D As shown. The second round of the second iteration can be similar to the first round of the first iteration, but with a second weight value instead of the first weight value. Similarly, each of the third weight value "-9" and the fourth weight value "3" can be processed in the third and fourth iterations respectively. Each of the third and fourth iterations can have two rounds, similar to the first and second iterations. At the end of the fourth iteration, the values ​​in the output map 1830 can form an output sub-feature map and can be stored in the accumulator buffer 1350.

[0219] Now go to Figures 19A to 19E An example of an input feature map 1900 in a sparse tensor computation unit 1500 according to some embodiments of the present disclosure is shown. It should be understood that the input feature map 1900 is merely an example and is not intended to be limiting in any way. The present disclosure can be used to process input feature maps of any size and containing any data values. The input feature map 1900 can be generated from an input image 205. Specifically, as... Figure 19A As shown, the input feature map 1900 may include multiple cells, each cell being a row (which extends in the X direction 1105; see also...) Figure 11 ) and column (extending in the Y direction 1110; see Figure 11 It is formed at the intersection of the input feature map 1900. Each cell in the multiple cells of the input feature map 1900 includes the data value that will be processed in the sparse tensor computation unit 1500.

[0220] Despite Figure 19ANot shown in the diagram, but in some embodiments, similar to input feature map 1100, input feature map 1900 may be filled with zero values ​​on all sides. Furthermore, in some embodiments, sub-feature maps may be created from input feature map 1900 (or from a filled feature map when padding is used). In some embodiments, input feature map 1900 may be segmented into sub-feature maps within partition blocks 210 or other components of accelerator 200, as described above. Figure 3A and Figure 3B As discussed in [the document]. For example, and as [the document discusses]. Figure 19A As shown, the input feature map 1900 can be divided into multiple sub-feature maps 1905A-1905P. In some embodiments, at least some of the sub-feature maps 1905A-1905P can be processed sequentially (e.g., one after another) in the sparse tensor computation unit 1500. In other embodiments, at least some of the sub-feature maps 1905A-1905P can be processed in parallel in multiple instances of the sparse tensor computation unit 1500. In some embodiments, the multiple sub-feature maps 1905A-1905P can be processed in the order shown by arrow 1910. Furthermore, within each sub-feature map 1905A-1905P, data values ​​can be processed in the order shown by arrow 1915. In other embodiments, the order in which the multiple sub-feature maps 1905A-1905P are processed and / or the order in which the data values ​​of each sub-feature map are processed can vary. In addition, in some implementations, only non-zero data values ​​can be input into the sparse tensor computation unit 1500.

[0221] Although each of the multiple sub-feature maps 1905A-1905P comprises two rows and two columns, and all of those sub-feature maps have the same size, it should be understood that in other embodiments, those sub-feature maps may have different sizes, may include different numbers of rows and columns, and the data values ​​within each of the multiple sub-feature maps 1905A-1905P may vary. Furthermore, depending on the number of multipliers 1505A-1505P in the sparse tensor computation unit 1500, multiple data values ​​of each of the multiple sub-feature maps 1505A-1505P can be processed in parallel. For illustrative purposes, the explanation is relative to sub-feature map 1905A. Figures 19B to 19E However, other sub-feature maps in multiple sub-feature maps 1905B-1905P can be handled in a similar manner. Furthermore, the description assumes that the number of multiple multipliers 1505A-1505P is three. Figures 19A to 19ESince sub-feature map 1905A contains only two non-zero data values, each iteration can include a single round. It should be understood that the number of rounds in each iteration can vary depending on the sub-feature map. For example, sub-feature map 1905B contains three non-zero data values, and since three multipliers are used in the current example, each iteration processing this sub-feature map can still include a single round. On the other hand, sub-feature map 1905F contains four non-zero data values. Therefore, with three multipliers, each iteration processing sub-feature map 1905F can include two rounds.

[0222] The data values ​​of sub-feature map 1905A can be combined with the weight values ​​of kernel matrix 1920. It should be understood that kernel matrix 1920 is merely an example and is not intended to be limiting in any way. Kernel matrix 1920 may assume other dimensions (e.g., the number of rows and columns may differ from those shown), and the values ​​within the kernel matrix may also vary. Furthermore, although each of the multiple sub-feature maps 1905A-1905P has the same number of rows and columns as kernel matrix 1920, in some embodiments, one or more of the multiple sub-feature maps may have a different number of rows and / or columns than the kernel matrix. The weight values ​​in kernel matrix 1920 can be applied in the specified order as indicated by arrow 1925. Furthermore, as described above, sparse tensor computation unit 1500 is configured to process multiple weight values ​​in each iteration. Therefore, multiple weight values ​​can be broadcast in each iteration depending on the number of non-zero data values ​​and the number of multipliers. This is explained under the assumption that two unique weight values ​​can be broadcast in each iteration. Figures 19A-19E Examples.

[0223] For example and such Figure 19B As shown, sub-feature map 1905A contains only two non-zero values. Therefore, only the first two of the three multipliers are needed in the first iteration to process the two non-zero data values. However, to improve performance and make full use of resources, the second weight value and the first non-zero data value can be broadcast to the third multiplier. For example, the data values ​​"9" and "-5" of sub-feature map 1905A can be input into multipliers 1505A and 1505B, respectively. Furthermore, the first weight value "-6" can be input into multipliers 1505A and 1505B. Since we have three multipliers, the first non-zero data value "9" is input into multiplier 1505C in the order shown by arrow 1915. Furthermore, the second weight value "6" is input into multiplier 1505C to begin the second iteration. Therefore, the second iteration overlaps with the first iteration.

[0224] Each multiplier in multipliers 1505A-1505C can also receive the index value of the data value received by each of those multipliers, such as... Figure 19B As shown. Furthermore, each multiplier in multipliers 1505A and 1505B receives the index value of the first weight value, and multiplier 1505C receives the index value of the second weight value. Based on the received index values ​​of the data values, the index values ​​of the first weight value and the second data value, each of those multipliers 1505A-1505C can compute the result index value of the location in the output sub-feature map where the result of the multiplication will be stored, as discussed above. The product result and the result index value can be transferred to MIN 1525. Since the result index values ​​computed in multipliers 1505A-1505C are different from each other, MIN 1525 simply passes the product result and the result index value to adder block 1535.

[0225] Addition block 1535 can perform the following additions:

[0226] s1 = b1 + b3

[0227] s2 = b2 + b1

[0228] s3 = b3 + b2

[0229] In the above formula, b1, b2, and b3 are the outputs from MIN 1525, and correspond to the product results generated by multipliers 1505A, 1505B, and 1505C, respectively. Figure 19B As further shown, the sum from adder 1535 and the result index value can be input into selection unit 1545, which includes multiple multiplexers. In some embodiments, selection unit 1545 may include three multiplexers to process the results from three multipliers.

[0230] The results and result index values ​​from the multiplexer in selection unit 1545 can be sent to accumulator 1555, which adds values ​​s1, s2, and s3 to those values ​​from previous iterations, as discussed above and as... Figure 19B As shown. Specifically, the result index value can be input into MIN 1575, which can read the value corresponding to those index values ​​from VAR 1565 and transfer those read values ​​to accumulator 1555. The result and result index value from the accumulator can be transferred to MIN 1560 and written to VAR 1565.

[0231] Figure 19CThe second iteration (or more precisely, the second half of the second iteration) overlaps with the second iteration. As discussed above, the second weight value and the data value "9" are processed during the first iteration (or the first half of the second iteration). In the second half of the second iteration, the remaining non-zero data value "-5" is input into multiplier 1505A along with the second weight value. Since there are three multipliers, the remaining two non-zero data values ​​are input into multipliers 1505B and 1505C along with the third weight value "-9". Thus, the second half of the second iteration and the third iteration occur in parallel. The processing of the second iteration is similar to that of the first iteration and therefore will not be described further. Similarly, the sub-feature map 1905A can be processed using the remaining weight values.

[0232] Figure 19D An example of a sub-feature map 1905C with a single non-zero data value is shown. In some implementations, with a single non-zero data value and three multipliers, three weight values ​​can be applied in a single iteration. For example, in the first iteration, the non-zero data value "-2" can be input into multiplier 1905A along with the first weight value "-6". With two remaining multipliers, the non-zero data value "-2" can also be input into multiplier 1905B along with the second weight value "6" and into multiplier 1905C along with the third weight value "-9". However, if the sparse tensor computation unit 1900 is configured to limit the number of unique weight values ​​to two in a particular iteration, only the first and second data weight values ​​can be processed in the first iteration. In this case, multiplier 1905C may not receive the non-zero data value "-2" and the third weight value. More precisely, multiplier 1905C can be in an idle state, such as... Figure 19D As shown. The processing performed in the first iteration using multipliers 1905A and 1905B can be done in a similar manner to that described above.

[0233] Figure 19E An example is shown where the resulting index value of one iteration overlaps with the overlapping half of the next iteration. For example, sub-feature map 1905J includes two non-zero values, which can be handled in a similar manner to that described above for sub-feature map 1905A. However, the resulting index value (5, 2) calculated for the product of data value "9" and the first weight value "-6" is the same as the resulting index value (5, 2) calculated for the product of data value "-9" and the second weight value "6". In this case, a merging operation can be performed as follows: Figure 20A and 20B As discussed in the article.

[0234] Figures 19A to 19EExamples can also be used to use Figure 14 The sparse tensor computation unit 1400 processes the input feature map 1900. The sparse tensor computation unit 1400 is designed for use with two unique weight values. Therefore, Figures 19A to 19E The example can also be applied to sparse tensor computation unit 1400. Except for the adder block 1535 and the selection unit 1545, sparse tensor computation unit 1400 is similar to sparse tensor computation unit 1500. Since MIN 1525 is similar to MIN 1425, data from MIN 1425 can be directly transferred to the accumulator 1430 in sparse tensor computation unit 1400, instead of MIN 1525 transferring data to the adder block 1535 in sparse tensor computation unit 1500. Both MIN 1425 and MIN 1525 can be used to avoid conflicts when two product values ​​have the same result index value. When such a conflict occurs, MIN 1425 and MIN 1525 apply a merge operation, which is then performed in the following... Figure 20A and 20B The discussion is ongoing.

[0235] Now go to Figure 20A and Figure 20B Examples of merging operations according to some embodiments of this disclosure are explained. The merging operation is described with respect to sparse tensor computation unit 1400, but this merging operation can be similarly implemented in sparse tensor computation unit 1500. As described above with respect to sub-feature map 1905J and... Figure 19E The index value (5, 2) resulting from the product of the data value "9" and the first weight value "-6" discussed is the same as the index value (5, 2) resulting from the product of the data value "-9" and the second weight value "6". A merge operation resolves conflicts in the resulting index values. The merge operation can be implemented by MIN 1425 in sparse tensor computation unit 1400 (or MIN 1525 in sparse tensor computation unit 1500). In some implementations, MIN 1425 (and MIN 1525) may include a FIFO (First-In-First-Out) buffer 2000 to resolve conflicts.

[0236] When MIN 1425 (or MIN 1525) receives result index values ​​from multiple multipliers 1405A-1405P, upon detecting a conflict, the MIN can transfer one of the product values ​​and its corresponding result index value to FIFO 2000. In some implementations, the product values ​​and corresponding result index values ​​transferred to FIFO 2000 can be based on the order in which the weight values ​​are being processed. For example, in the example of sub-feature map 1905J, the product value corresponding to the second weight value can be transferred to FIFO 2000 instead of the product value corresponding to the first weight value. In some implementations, MIN 1425 (and MIN 1525) can be programmed with the product values ​​to be transferred to FIFO 2000. If more than two conflicts exist (e.g., more than two product values ​​have the same result index value), all conflicting product values ​​except for one conflicting product value can be sent to FIFO 2000.

[0237] When transferring the product value to FIFO 2000, the remaining product value and its corresponding result index value can be transferred to either accumulator 1430 (in sparse tensor computation unit 1400) or adder block 1535 (in sparse tensor computation unit 1500). Therefore, when three values ​​are processed in parallel in multiple multipliers 1405A-1405C and a conflict exists between two product values, one of the conflicting product values ​​is stored in FIFO 2000, and only the two product values ​​are further processed. The product value stored in FIFO 2000 is processed in the next round / next iteration. Thus, as... Figure 20B As shown, in the next round / next iteration of processing the sub-feature map 1905J, only two data values ​​are input into multiple multipliers 1405A-1405P to generate two product values. The two product values ​​are sent to MIN 1425, and if there is no conflict between the product value received by MIN and the product value stored in FIFO 2000, MIN outputs three product values, as shown. Figure 20B As shown. Then, the remaining post-processing proceeds as usual, as discussed above.

[0238] The various exemplary logic blocks and modules described in conjunction with the embodiments disclosed herein can be implemented or executed by a machine designed to perform the functions described herein, such as a general-purpose processor device, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic components, discrete hardware components, or any combination thereof. A control processor can synthesize a model of an FPGA. For example, a control processor can synthesize a model of programmable logic gates to implement tensor arrays and / or pixel arrays. A control channel can synthesize a model to connect tensor arrays and / or pixel arrays on an FPGA, a reconfigurable chip, and / or die, etc. A general-purpose processor device can be a microprocessor, but alternatively, a processor device can be a controller, a microcontroller, or a state machine, a combination thereof, etc. A processor device can include circuitry configured to process computer-executable instructions. In another embodiment, a processor device includes an FPGA or other programmable device that performs logic operations without processing computer-executable instructions. A processor device can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Although this document describes primarily in relation to digital technologies, processor devices may also primarily include analog components. For example, some or all of the algorithms described herein may be implemented in analog circuits or mixed analog and digital circuits. The computing environment may include any type of computer system, including but not limited to, for example, microprocessor-based computer systems, mainframe computers, digital signal processors, portable computing devices, device controllers, or computing engines within appliances.

[0239] Elements of the methods, processes, routines, or algorithms described in conjunction with the embodiments disclosed herein may be directly embodied in hardware, in software modules executed by a processor device, or a combination of both. Software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of non-transitory computer-readable storage medium. Exemplary storage media may be coupled to a processor device such that the processor device can read information from and write information to the storage medium. Alternatively, the storage medium may be a component of the processor device. The processor device and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor device and storage medium may reside as discrete components in the user terminal.

[0240] The topics described herein sometimes illustrate different components contained within or connected to different other components. It should be understood that such depicted architectures are merely exemplary, and in fact, many other architectures can be implemented to achieve the same functionality. Conceptually, any arrangement of components used to achieve the same functionality is effectively “associated” to achieve the desired function. Therefore, any two components combined herein to achieve a particular function can be considered “associated” with each other to achieve the desired function, regardless of the architecture or intermediate components. Similarly, any two components so associated can also be considered “operably connected” or “operably coupled” to each other to achieve the desired function, and any two components that can be so associated can also be considered “operably coupled” to each other to achieve the desired function. Specific examples of operably coupled components include, but are not limited to, physically matchable and / or physically interactable components and / or wirelessly interactable and / or logically interactable components.

[0241] Regarding the use of any plural and / or singular terms herein, those skilled in the art can convert from plural to singular and / or from singular to plural depending on the context and / or application. For clarity, various singular / plural transformations may be explicitly described herein.

[0242] Those skilled in the art will understand that, generally, the terms used herein, particularly in the appended claims (e.g., the body of the appended claims), typically mean "open-ended" terms (e.g., the term "comprising" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," the term "comprising" should be interpreted as "including but not limited to," etc.). Those skilled in the art will also understand that if there is an intent to introduce a specific number of introduced claim statements, this intent will be explicitly stated in the claims, and if no such statements are present, such an intent does not exist. For example, to aid understanding, the appended claims may contain introductory phrases such as "at least one" and "one or more" to introduce claim statements. However, the use of such phrases should not be construed as implying that a claim statement introduced by the indefinite article "a" or "an" limits any particular claim containing such an introduced claim statement to an invention containing only one such statement, even when the same claim includes the introductory phrase "one or more" or "at least one" and indefinite articles such as "a" or "an" (e.g., "a" and / or "an" should generally be interpreted as meaning "at least one" or "one or more"); the same applies to the use of definite articles for introducing claim statements. Furthermore, even when a specific number of introduced claim statements is explicitly stated, those skilled in the art will recognize that such statements should generally be interpreted as meaning at least the number of stated statements (e.g., in the absence of other modifiers, the unmodified statement "two statements" generally means at least two statements, or two or more statements). Furthermore, in cases where conventional conventions such as "at least one of A, B, and C" are used, this construction generally means in the sense that a person skilled in the art would understand from conventional conventions (e.g., "a system having at least one of A, B, and C" includes, but is not limited to, systems having A alone, having B alone, having C alone, having both A and B, having both A and C, having both B and C, and / or having both A, B, and C, etc.). In cases where conventional conventions such as "at least one of A, B, or C" are used, this construction generally means in the sense that a person skilled in the art would understand from conventional conventions (e.g., "a system having at least one of A, B, or C" includes, but is not limited to, systems having A alone, having B alone, having C alone, having both A and B, having both A and C, having both B and C, and / or having both A, B, and C, etc.). Those skilled in the art will further understand that any separate words and / or phrases presenting two or more alternative terms, whether in the specification, claims or drawings, should be understood to envision the possibility of including one, any, or both of the terms.For example, the phrase “A or B” will be understood to include the possibility of “A” or “B” or “A and B”. Furthermore, unless otherwise specified, the words “approximately,” “around,” “basically,” etc., mean plus or minus ten percent.

[0243] The foregoing description of exemplary embodiments has been presented for purposes of illustration and description. It is not intended to be exhaustive or limiting in terms of the precise forms disclosed, and modifications and variations are possible, or may be derived, from practice of the disclosed embodiments in light of the foregoing teachings. The scope of the invention is intended to be defined by the appended claims and their equivalents.

Claims

1. A system for performing computations on a sparse input feature map divided into a plurality of sub-feature maps, each sub-feature map having a respective sparsity according to a percentage of zero values in the sub-feature map, such that a sparsity of each sub-feature map of the plurality of sub-feature maps that satisfies is within a predetermined threshold range, the system comprising: a multiplication circuit comprising a plurality of multipliers, each of the plurality of multipliers configured to receive a data value and a weight value to generate a product value in a convolution operation of a machine learning application, the data value being part of a sub-feature map of a plurality of sub-feature maps generated from an input feature map; an accumulator configured to receive the product value from each of the plurality of multipliers; and a register bank configured to store an output of the convolution operation, wherein the accumulator is further configured to receive a portion of values stored in the register bank and combine the received portion of values with the product value to generate a combined value; and wherein the register bank is further configured to replace the portion of values with the combined value.

2. The system of claim 1, wherein the register bank comprises a plurality of row registers configured to shift in a row direction and a plurality of column registers configured to shift in a column direction.

3. The system of any one of claims 1-2, wherein combining the received portion of values with the product value comprises adding each of the received portion of values to a corresponding product value of the product value.

4. The system of any one of claims 1-2, further comprising a reconfigurable tree adder configured to receive the product value and combine the product value into groups.

5. The system of any one of claims 1-2, wherein the register bank is configured to shift a subset of values from a previous iteration by one position and send the shifted subset of values to the accumulator.

6. A system for performing computations on a sparse input feature map divided into a plurality of sub-feature maps, each sub-feature map having a respective sparsity according to a percentage of zero values in the sub-feature map, such that a sparsity of each sub-feature map of the plurality of sub-feature maps that satisfies is within a predetermined threshold range, the system comprising: a multiplication circuit comprising a plurality of multipliers, each of the plurality of multipliers configured to receive a data value and a weight value to generate a product value in a convolution operation of a machine learning application; an accumulator configured to receive the product value from each of the plurality of multipliers; and a register bank configured to store an output of the convolution operation, wherein the accumulator is further configured to receive a portion of values stored in the register bank and combine the received portion of values with the product value to generate a combined value; and wherein the register bank is further configured to replace the portion of values with the combined value. wherein the accumulator is further configured to receive a portion of the values stored in the register bank and combine the received portion of values with the product values to generate a combined value; and wherein the register bank is further configured to replace the portion of values with the combined value; and a first multi-stage interconnect network configured to receive the combined value from the accumulator.

7. The system of claim 6, wherein the first multi-stage interconnect network is configured to sort the combined value and write the sorted combined value into a vector accumulator register.

8. The system of claim 7, further comprising a second multi-stage interconnect network configured to read a subset of values from the vector accumulator register and send the subset of values to the accumulator.

9. The system of claim 7, wherein the vector accumulator register is further configured to receive the portion of values from the register bank prior to sending the portion of values to the accumulator.

10. The system of claim 6, further comprising a third multi-stage interconnect network configured to receive the product values from the plurality of multipliers and send at least some of the product values to the accumulator based on an index value of each of the product values.

11. A method for performing a computation on a sparse input feature map divided into a plurality of sub-feature maps, each sub-feature map having a respective sparsity according to a percentage of zero values in the sub-feature map, such that a sparsity of each of the plurality of sub-feature maps satisfying is within a predetermined threshold range, the method comprising: inputting, by a processor in a machine learning application, data values and weight values into each of a plurality of multipliers to generate a plurality of product values in each of a plurality of iterations of a convolution operation; combining, by the processor in each of the plurality of iterations, each of the plurality of product values with one of a plurality of accumulator values in an accumulator to generate a plurality of combined values, wherein the plurality of accumulator values for a current iteration are received from a register bank and obtained by shifting a subset of values in the register bank one position after a previous iteration; and replacing, by the processor in each of the plurality of iterations, the plurality of accumulator values with the plurality of combined values in the register bank.

12. The method of claim 11, wherein values in the register bank after a last iteration of the plurality of iterations provide an output of the convolution operation on an input sub-feature map generated from an input feature map.

13. The method of any of claims 11-12, wherein each of the plurality of multipliers receives the same weight value.

14. The method of any of claims 11-12, wherein at least one of the plurality of multipliers receives a different weight value than the weight values received by the remaining multipliers of the plurality of multipliers.

15. The method of any of claims 11-12, further comprising receiving the combined values from the accumulators in a first multi-level interconnection network.

16. The method of claim 11, further comprising shifting values in the register file by the processor after a last iteration of the multiple iterations to obtain an output sub-feature map.

17. A non-transitory computer-readable medium comprising computer-readable instructions stored thereon, which, when executed by a processor associated with a machine learning application, cause the processor to: divide an input feature map into a plurality of sub-feature maps; input each sub-feature map of the plurality of sub-feature maps into a tensor computation unit of a plurality of tensor computation units to generate an output sub-feature map, wherein generation of the output sub-feature map for a first sub-feature map of the plurality of sub-feature maps comprises: inputting a plurality of data values of the first sub-feature map into a plurality of multipliers of a first tensor computation unit of the plurality of tensor computation units; inputting a weight value into the plurality of multipliers for generating a plurality of product values; combining each product value of the plurality of product values and one of previously computed product values to obtain a plurality of combined values; and shifting the plurality of combined values to obtain the output sub-feature map for the first sub-feature map; and combining the output sub-feature maps from each of the plurality of tensor computation units to obtain an output feature map; and wherein the computation is performed on a sparse input feature map divided into a plurality of sub-feature maps, each sub-feature map having a respective sparsity according to a percentage of zero values in the sub-feature map, such that a sparsity of each sub-feature map of the plurality of sub-feature maps satisfying is within a predetermined threshold range.

18. The non-transitory computer-readable medium of claim 17, further comprising performing a rectified linear unit operation and a pooling operation on the shifted plurality of combined values to obtain the output sub-feature map.

19. The non-transitory computer-readable medium of claim 17, further comprising compressing the output sub-feature map before combining to obtain the output feature map.

20. The non-transitory computer-readable medium of claim 18, wherein each of the plurality of data values input into the plurality of multipliers is a non-zero value, and wherein the weight value is a non-zero value. ​ ​

Images