A spatially sparse neural network accelerator for multi-dimensional visual analytics

The SSpNNA addresses inefficiencies in 3D visual analytics by employing a spatially sparse neural network accelerator with WAVES and SyMAC, achieving up to 90% resource utilization and efficient processing of sparse and dense data.

JP7581579B2Active Publication Date: 2024-11-13INTEL CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2021131382
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-12-22
Filing Date
2021-08-11
Publication Date
2024-11-13
Estimated Expiration
2041-08-11

AI Technical Summary

Technical Problem

Existing neural network accelerators struggle to efficiently process 3D visual analytics due to high computational and memory requirements, particularly in scenarios involving spatial sparsity, leading to inefficiencies in real-time and energy-efficient deployment.

Method used

A spatially sparse neural network accelerator (SSpNNA) that utilizes a Weight plane based Active Voxel Execution Scheduler (WAVES) and Systolic and Multicast based MAC Computation (SyMAC) to process 3D sparse data efficiently, dynamically allocating resources and reducing memory requirements through innovative microarchitectural enhancements.

Benefits of technology

The SSpNNA significantly reduces computational and memory requirements by up to 90% utilization, supporting both sparse and dense workloads with enhanced hardware efficiency and scalability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581579000001
    Figure 0007581579000001
  • Figure 0007581579000002
    Figure 0007581579000002
  • Figure 0007581579000003
    Figure 0007581579000003
Patent Text Reader

Abstract

To provide a computing system for improving efficiency of sparse convolution processing using a spatially sparse neural network accelerator for multi-dimension visual analytics, a program, a method and a semiconductor device.SOLUTION: A method includes: decoding data via an instruction that indicates the number of rulebooks to be processed, an input feature size, an output feature size, and a plurality of feature map base addresses; rearranging spatially distributed voxel output feature maps in the decoded data based on weight planes; and performing a channel-wise multiply-accumulate (MAC) operation on the rearranged spatially distributed voxel output feature maps to obtain an output. The channel-wise MAC operation is performed as partial accumulations by a plurality of processing elements.SELECTED DRAWING: Figure 8
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to Indian Provisional Patent Application No. 202041042682 filed on October 1, 2020.

[0002] FIELD OF THE DISCLOSURE Embodiments relate generally to neural network accelerators and, more particularly, to spatially sparse neural network accelerators for multi-dimensional visual analytics. [Background technology]

[0003] Semantic segmentation and completion of real-world scenes are fundamental primitives of three-dimensional (3D) vision that are widely used in high-level applications such as robotics, medical imaging, autonomous driving and navigation, etc. However, the computational and memory requirements of 3D visual analytics can grow with the complexity of the third dimension at voxel resolution, making real-time energy-efficient deployment a real-time obstacle. [Brief description of the drawings]

[0004] Various advantages of the embodiments will become apparent to those skilled in the art upon reading the following specification and appended claims, and upon review of the following drawings.

[0005] [Figure 1] FIG. 2 illustrates an example of a rulebook line format, according to an embodiment.

[0006] [Diagram 2] FIG. 1 illustrates an example of a rulebook creation procedure, according to an embodiment.

[0007] [Figure 3A] FIG. 13 illustrates examples of different tile options, according to an embodiment. [Figure 3B]FIG. 13 illustrates examples of different tile options, according to an embodiment. [Figure 3C] FIG. 13 illustrates examples of different tile options, according to an embodiment.

[0008] [Figure 4] FIG. 2 is a block diagram of an example of a hardware accelerator (HWA) and memory layout, according to an embodiment.

[0009] [Diagram 5] FIG. 13 is a block diagram of an example weight plane active voxel execution scheduler, according to an embodiment.

[0010] [Figure 6] FIG. 2 is a comparative diagram illustrating a fixed resource based index queue and a linked list based index queue according to an embodiment.

[0011] [Figure 7] FIG. 2 is a block diagram of an example of systolic and multicast-based multiply-accumulate (MAC) hardware, according to an embodiment.

[0012] [Figure 8] 1 is a flowchart of an example method of operating an enhanced performance computing system, according to an embodiment.

[0013] [Figure 9] 4 is a flowchart of an example of a more detailed method of operating an enhanced performance computing system, according to an embodiment.

[0014] [Figure 10] 1 is a flowchart of an example method for managing feature storage, according to an embodiment.

[0015] [Figure 11] FIG. 1 is a block diagram of an example of an enhanced performance computing system, according to an embodiment.

[0016] [Figure 12] FIG. 1 illustrates an example of a semiconductor device according to an embodiment.

[0017] [Figure 13] FIG. 2 is a block diagram of an example processor, according to an embodiment.

[0018] [Figure 14] FIG. 1 is a block diagram of an example multi-processor based computing system, according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] In 3D visual analytics scenarios, it becomes increasingly important to exploit data sparsity as much as possible to reduce the computational resources involved in data processing. Traditional implementations of convolutional networks and corresponding accelerators can be optimized for data that resides on a dense grid.

[0020] Commercial deep neural network (DNN or DeNN) accelerators such as Tensor Processing Units (TPU, GOOGLE) and Tensor Cores (NVIDIA) may use dense two-dimensional arrays optimized for highly regular data flows. Commercially available sparse accelerators may specialize in two-dimensional (2D) arrays and accelerate the determination of element overlaps. However, sparse convolution processing using such accelerators may be highly inefficient.

[0021] For example, traditional applications may be mapped onto a central processing unit (CPU, e.g., a host processor) and a graphics processing unit (GPU, e.g., a graphics processor), but the performance may still be far below the requirements for real-time use. Indeed, attempts to address spatial sparsity may be compiler extensions for CPUs and GPUs. One traditional approach may provide a high-level interface to efficient data structures for spatially sparse data and generate code for sparse computations on CPUs and GPUs. Another traditional approach may generate a CPU implementation of sparse convolutions using data layouts and kernel templates with improved loop tiling and vectorization.

[0022] These traditional convolutional network implementations may be optimized for data stored in a dense grid and cannot efficiently handle sparse data. More recently, several convolutional network implementations have been presented that can be adapted to operate efficiently on sparse data. Mathematically, some of these implementations are identical to regular convolutional networks, but they require less computational resources in terms of floating-point operations (FLOPs) per second and / or memory. Previous operations use sparse versions of the "im2col" operation that restricts computation and storage to "active" sites, or use voting procedures that remove unnecessary multiplications by zero.

[0023] Other solutions may use a fixed-function tile matrix multiply (TMUL) unit that uses a zero-detection block to process sparse data and skips operations on entry to a processing element (PE) stage. For storage, solutions may use sparse compression algorithms such as Compression Sparse Row (CSR) and / or Compression Sparse Column (CSC). Solutions may also rely on zero-detection logic to process sparse data. However, solutions may cater to 2D sparsity and do not match the real-world 3D spatial sparsity, which is a fundamental characteristic of 3D visual understanding applications. In short, no hardware accelerator has been proposed to accelerate basic 3D sparse convolution operations.

[0024] Accordingly, a key drawback of previous sparse implementations of convolutional networks is that they expand sparse data at every layer by applying full convolutions. The use of zero detection logic can result in underutilization of the hardware, which is important considering, for example, that each zero detected in an array can cause an underutilization of (1 / array size)%.

[0025] For such 3D vision applications, the embodiments provide an efficient and scalable hardware solution without incurring significant area and power penalties. More specifically, the embodiments include a Spatially Sparse Neural Network Accelerator (SSpNNA) that can decode and efficiently process 3D sparse data (e.g., the relationship between input feature maps / IFMs and output feature maps / OFMs corresponding to weight planes) encoded in a rulebook format. The techniques described herein provide an end-to-end hardware solution for N-dimensional visual analytics. The embodiments also include new instructions to drive the hardware. The hardware may include two main blocks: 1) Weight plane based Active Voxel Execution Scheduler (WAVES), which performs formatting to rearrange spatially distributed voxel OFMs, and 2) Systolic and Multicast based MAC Computation (SyMAC), which performs per-channel computation and collection of output elements. The proposed instructions may provide all required pointers to the SSpNNA for seamless processing. An advantage of the SSpNNA accelerator architecture is that it can significantly reduce the computational and memory requirements of running 3D visual analytics applications.

[0026] FIG. 1 illustrates a rulebook line format 20. In the illustrated example, a first rulebook line 22 specifies a weight mask ("101011---110") for a first input channel index (e.g., "IC-1," corresponding to the first input voxel in the input receptive field / IRF) and a set of output feature indices (e.g., corresponding to valid output voxels in the output response field / ORF). The position of a 1 in the weight mask specifies the weight used to determine the contribution of the first input voxel to each output feature. For example, a 1 value in the leftmost position indicates that weight w1 is used to determine the contribution of IC-1 to the leftmost output feature index ("OC6"), a 1 value in the third position from the left indicates that weight w3 is used for output feature index "OC9," etc. Similarly, the second rule book 24 may specify a weight mask ("001010--001") and a set of output feature indexes for a second input channel index ("IC-2"), and line 26 of the third rule book may specify a weight mask ("100010--000") and a set of output feature indexes for a third input channel index ("IC-3").

[0027] In the example shown, there is asymmetry across weights because, for example, weight w1 is not used uniformly across all three rulebook lines 22, 24, 26. Additionally, there is asymmetry across rulebook lines 22, 24, 26 because, for example, the number of output features in the first rulebook line 22 is different from the number of output features in the second rulebook 24.

[0028] The SSpNNA described herein may accelerate sparse processing of N-dimensions (e.g., variables with dimensions greater than 2) compared to conventional solutions and may increase the overall utilization of computational resources to approximately 90%. Along with supporting sparse processing, the hardware described herein may also operate efficiently on dense workloads. For example, for a ScanNet (SCN) workload with 39 layers, the utilization of the hardware ranges from 78.8% to 98.7%, while achieving an average utilization of approximately 93.20%. The hardware may also support dense neural network (NN) workloads by processing all bits in a bitmask (e.g., weight mask) that are set. This higher utilization is achieved through microarchitectural enhancements summarized as follows:

[0029] [WAVES]

[0030] N-dimensional spatially sparse convolution - defined with microarchitectures processing weight planes hierarchically, selecting a number of planes to be processed at a time based on functional block area budget. This feature enables designs to operate beyond fixed convolutions. For example, for hardware with 32 weight planes, 3x3x3 convolutions (27 weight planes) may be processed together, and 5x5x5 convolutions (125 weight planes) may be processed iteratively.

[0031] Dynamic resource allocation - Dynamic allocation of smaller chunks of memory based on sparsity can help accommodate approximately 1.5x more rulebook (RB) lines (rb lines) instead of storage mapped as a fixed resource per weight plane.

[0032] Input Channel (IC) Storage with Index to Reduce Duplicate Entries - Input to output (i2o) or output to input (o2i) feature mappings can be stored as pairs in per weight storage. To reduce storage width, features (e.g., 32-bit floating point / FP numbers) can be stored in a static buffer and corresponding indexes (e.g., 8-bit) can be stored in data storage (e.g., index queue), which helps reduce feature storage by about 75%.

[0033] Enabling different types of rulebooks (i2o, o2i) using the same index buffer - The type of rulebook (e.g., i2o type and o2i type) can be dynamically selected based on the application. For example, the index-based storage techniques described herein enable both types to be scheduled by swapping data at the output of the index queue (e.g., keeping the overall WAVES design the same for both RB types).

[0034] [SyMAC]

[0035] 3D Sparse Convolution Microarchitecture - Embodiments include a microarchitecture that shares weight data as a dynamic systolic array (e.g., a homogenous network of tightly coupled data processing units DPUs, cells, and / or nodes). Furthermore, input features can be multicast to multiple processing elements (e.g., performing addition of dot product results) and output features can be partially added. These enhancements make the technique applicable to different tile dimensions.

[0036] Recirculating Buffer for Maximum Reuse of IC Data Buffer - To increase reuse of IFMs across multiple OFMs, the IC data buffer may be implemented as a recirculating buffer that provides IC values ​​to multiple PEs in the DeNN.

[0037] Feature collision detection and summing - Caching of output features and local summing to reduce Level 1 (L1) cache bandwidth.

[0038] [Implementation details]

[0039] FIG. 2 illustrates a procedure 30 for generating drb lines of a rulebook 32. In an embodiment, the encoding of the rulebook 32 can be of two types - i2o (input to output relationship) or o2i (output to input relationship). In the illustrated example, the rulebook 32 is encoded as an i2o data structure. Tiles are processed by which subsets of input or output voxels and input or output channels (e.g., each subset known as a "tile") can be grouped and then processed in stages to save memory and data access. A tile can be defined by a set of parameters drb, dic, and doc, where drb refers to the number of rb lines in the tile, and dic and doc refer to the number of input and output channels in the tile, respectively. An IFM tile is composed of di input voxels, where each voxel has a dic number of elements (e.g., channels). Similarly, an OFM tile contains do output voxels and a doc element (e.g., channel) per voxel. For the i2o rulebook 32, drb is equal to di and do may vary across tiles based on sparsity. For the o2i rulebook, drb is equal to do and di may vary across tiles based on sparsity.

[0040] For illustrative purposes, a 2D sparse convolution with a 3x3 filter may be used. An input at location (x;y) is stored with index i=10 and contributes to the set of active outputs ORF10={7;5;10;8} with weights {w1;w2;w4;w6}. ORF10={oj} and the corresponding weight bitmask form the first line in the i2o rulebook 32. Similarly, each line in the o2i rulebook 32 is a filter for each output index o n IRF along with a bitmask of weights for n= {ij}. The shaded boxes indicate index overlaps between drb lines.

[0041] 3A-3C show that a key requirement for a hardware accelerator (HWA) is to operate in tiles, where the relationship between the IFM and OFM is defined using lines from the rulebook, and the sparsity across weight planes is encoded into the weight plane's mask. Performing efficient computational operations with these inter-tile transformations involves fetching variable data per dimension, but takes full advantage of internal data locality.

[0042] More specifically, FIG. 3A illustrates a first tile option 40 in which OC is shared across tiles. Accordingly, there may be fewer rb lines with more dic and minimal doc. FIG. 3B illustrates a second tile option 42 in which IC is shared across tiles. In such a case, there may be fewer rb lines with more doc and minimal dic. FIG. 3C illustrates a third tile option 44 in which weights are shared across tiles. Accordingly, there may be more rb lines with fewer dic and doc. The order of tiles shown may be dynamically selected by software based on application data. Additionally, HWA may be capable of accelerating under all options 40, 42, 44 with limited memory bandwidth.

[0043] As already noted, the Spatially Sparse Neural Network Accelerator (SSpNNA) hardware accelerator may include two main blocks: (a) WAVES-weight plane based active voxel execution scheduler, and (b) SyMAC-systolic and multicast based MAC computation. In an embodiment, SSpNNA automatically combines systolic and broadcast approaches, but adds partial data locally.

[0044] FIG. 4 shows the HWA 50 and memory layout 52 of the SSpNNA. In the example shown, a configuration and control block 54 controls the overall execution, and a memory arbiter 56 arbitrates memory requests and handles the interface protocol. A global event controller (not shown) may load the L1 cache and configure the HWA 50 based on the newly proposed instructions, and then start the execution. At startup, the WAVES scheduler 58 starts fetching RB headers and performs a workload reformatting according to the weight plane. The WAVES scheduler 58 may schedule operations to the SyMAC computation block 60, which performs per-channel computations and collection of output elements. The tile data mechanism is in the L1 memory, which may be used as a scratchpad, for example, with a 16B interface. Considering the case of i2o RBs, the RB header mask and the O index (output feature index / Idx) of the header give an index into the OFM index and the corresponding list of RB weight masks. The index into the IFM is the index of the weight mask field, which may be stored linearly. The OFM index in the RB header provides an index to the OFM element data stored in the OFM data field. The number of indices in the OFM data field for each RB can be decoded from the number of ones in the corresponding weight mask. In an embodiment, these elements are packed tightly together to utilize all memory bits. The base addresses of the elements can be encoded into the new instruction and decoded accordingly by the hardware reading the corresponding value. The IFM data, weight data, and OFM data can be stored in any order, provided that the RB header captures the index accurately. In one example, the elements are packed into multiples of four to fit into the bandwidth of the memory port.

[0045] FIG. 5 shows a WAVES scheduler 70 that operates based on the output of a rulebook creation procedure that encodes the relationship between the IFM and the OFM corresponding to the weight plane. As already explained, the WAVES scheduler 70 can be easily replaced by the WAVES scheduler 58 (FIG. 4). In an embodiment, the WAVES scheduler 70 includes an RB header (HDR) processor 72 (72a-72c), an HDR format 74 (74a-74c), and a linked list buffer 76 (76a-76d). The operation of the WAVES scheduler 70 can start with a first fetch block 72a that fetches mask and O-Idx data, which provides a pointer to the list of OFM indexes and OFM distribution per weight. Based on the number of 1s in the weight mask calculated by the mask analyzer 72b, the respective number of OFM indexes is fetched by the second fetch block 72c.

[0046] To meet memory read latency, fetched OFM indexes may be stored in a FIFO (first in first out, e.g., a static buffer) 74c, and corresponding header information may be stored in a FIFO 74a (e.g., a static buffer). In one example, a tuple formation block 74b combines multiple IFM-OFM pairs that share the same weight plane. To meet a memory bandwidth of four FP (Floating Point) elements per cycle, the tuple formation block 74b may generate four tuples per cycle. The illustrated linked list buffer 76 has index queues 76a, 76c, which hold tuples before scheduling for computation and output via a multiplexer 76d. For example, the index queues 76a, 76c are duplicated to hide the header formation logic, such that the first index queue 76a is active and scheduling computation workloads while the second index queue 77c is collecting formatted rule books (e.g., vice versa). The index queues 76a, 76c may be controlled by a queue controller 76b.

[0047] FIG. 6 illustrates a linked list data structure 80 for a scheduler, such as the WAVES scheduler 70 (FIG. 5). In the illustrated example, the fixed resource-based index-Q 82 includes wavy lines that indicate a dynamic distribution of active voxels within the receptive field. Although the filter characteristics lead to symmetry around the weight plane 13, the SSpNNA may support any distribution. If the mapping were done via fixed hardware resources per plane, the vertical lines in the fixed resource-based index-Q 82 would represent underutilization as an effect of sparsity in the workload. In the SSpNNA, a linked list-based storage approach is applied to dynamically allocate resources and avoid hardware underutilization. Thus, the linked list data structure 80 uses a linked list-based weight plane allocation. To further reduce storage, the IFM index is stored in an IFM index buffer 84, and an index to that information is stored in a pointer table 86.

[0048] By allocating more resources to planes with more active adjacent voxels, it may be possible for 1.5-2 times more rulebook pencils to be accommodated in the same size memory inside the SSpNNA. As already noted, the input-to-output (i2o) and output-to-input (o2i) rulebook types may be dynamically selected based on the application. The index-based storage mechanism shown enables both rulebook types to be scheduled by exchanging data at the output of the index queues 76a, 76c (FIG. 5), keeping the overall WAVES design the same for both RB types. For N-dimensional problems, WAVES may operate with a fixed tile size of the weight plane and still be able to support the full end-to-end requirements. Furthermore, there is no degradation in utilization for dense workloads (e.g., all bits set in the weight bitmask), and WAVES supports the solution without any modifications.

[0049] FIG. 7 shows a SyMAC computation block 90 capable of operating on different systolic groups 92. As already explained, the SyMAC computation block 90 can easily be replaced by the SyMAC computation block 60 (FIG. 4). In an embodiment, the selection of the systolic group is done dynamically, and the scheduler always tries to form groups with larger DeNNs to reduce the weight demand on L1. At the boundary of a weight plane where not enough tuples are available to form a larger systolic group, the next weight plane is scheduled as another group. All processing elements (PEs) in the DeNN block 94 operate in series on the same IFM, and each weight performs four 32-bit floating point MACs as a pipelined operation to generate partial OFM data. All data buses can be, for example, 128 bits wide to accommodate four IEEE 754 full floating point numbers. In an embodiment, each DeNN block 94 has a DeNN-Q that uses a queue to buffer data before resources are ready to send. Each DeNN-Q may also schedule memory fetches of IFM and weight data. The IC data buffer (e.g., recirculating buffer) and weight (WT) block may include logic to place memory requests and buffer data before scheduling to PEs. As the OC (output channel) value increases, the IFM data in the IC data buffer may be reused multiple times, and as the IC (input channel) value increases, the percentage of local summation of partial data increases as PEs operate with the same OFM. These modes may be automatically enabled based on the IC and OC values. These design choices increase internal data reuse and reduce L1 accesses.

[0050] The Addition (ACC) OFM block 96 may add partial OFMs generated from multiple DeNNs, perform a tag search to find overlapping OFMs, and locally merge the overlapping OFMs. In an embodiment, the ACC OFM block 96 also requests the associated OFMs from memory to be merged with the generated OFM. A four DeNN configuration with four PEs per DeNN computes four elements per PE per cycle, allowing the SSpNNA to support 64 MUL operations per cycle. Changing the SSpNNA configuration to eight DeNNs and operating with two systolic groups of four DeNNs doubles the performance to 128 MUL operations per cycle, respectively, without requiring any additional memory ports for weights.

[0051] SSpNNA accelerates N-dimensional sparse processing compared to any available solution, boosting the overall utilization of computational resources to about 90%. This HWA can be used as a standalone accelerator or co-processor, where the following instructions can be used to drive the SSpNNA hardware:

[0052] The new instruction is SSXRBLNIFMOF Spatial Sparse with 'X' Rule Book lines having 'N' input Feature and 'M' Output Feature, where 'X' indicates the number of rule book lines to be processed and 'N' and 'M' represent the input feature size and output feature size, respectively. For example, if there are 32 rule book lines to be processed with 16 input and output feature maps, the instruction is SS32RBL16IF16OF. The size of the IFM and OFM can remain the same for the entire rule book and can be a multiple of 4 to fit the memory bandwidth.

[0053] In an embodiment, the format of the instruction is SSXRBLNIFMOF tsrcdest, tsrc1, tsrc2, tsrc3, where tsrcdest refers to the OFM base address for reading and writing back the partial / processed data, tsrc1 represents the base address of the line of the rulebook, and tsrc2 and tsrc3 represent the base addresses of the IFM and OFM, respectively.

[0054] 8 illustrates a method 100 for operating a performance enhancing computing system. As previously described, the method 100 may be generally implemented in a hardware accelerator, such as, for example, the HWA 50 (FIG. 4). More specifically, the method 100 may be implemented in one or more modules, as a set of logic instructions stored in a machine or computer readable storage medium, such as a random access memory (RAM), a read only memory (ROM), a programmable ROM (PROM), firmware, a flash memory, or the like; configurable logic, such as, for example, a programmable logic array (PLA), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), or the like; fixed function logic hardware, such as, for example, an application specific integrated circuit (ASIC), using circuit technology, such as complementary metal oxide semiconductor (CMOS) or transistor-transistor logic (TTL) technology, or any combination thereof.

[0055] For example, computer program code for performing the operations illustrated in method 100 may be written in any combination of one or more programming languages, including object-oriented programming languages, such as JAVA®, SMALLTALK®, C++, or the like, and conventional procedural programming languages, such as the “C” programming language or a similar programming language. Additionally, the logic instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state configuration data, configuration data for integrated circuits, state information for personalizing electronic circuits, and / or other structural components specific to hardware (e.g., a host processor, central processing unit / CPU, microcontroller, etc.).

[0056] The illustrated processing block 102 decodes the data with instructions indicating the number of rulebooks to be processed, the input feature size, the output feature size, and a number of feature map base addresses. In an embodiment, block 104 rearranges the spatially distributed voxel OFM in the decoded data based on the weight plane. In one example, block 104 arranges the OFM based on a least significant bit (LSB) hash of the OFM address. Thus, even if the processing elements are executing in any order, once the next block arrives, the OFM is first rearranged based on the hash value (e.g., LSB bit) and then based on the corresponding channel to which the OFM is added.

[0057] Block 106 performs a per-channel MAC operation on the rearranged spatially distributed voxel OFMs to obtain an output, where the per-channel MAC operation is performed as a partial summation by multiple processing elements in the computing system. In an embodiment, block 106 allocates multiple processing elements based on the sparsity of the data. Additionally, the per-channel MAC operation may identify overlapping OFMs (e.g., using tag search) and may merge the overlapping OFMs locally (e.g., rather than globally). The illustrated method 100 enhances performance, at least to the extent that the instructions facilitate more seamless and efficient execution, reduce computation and memory requirements by rearranging the spatially distributed voxels based on weight planes, and / or increase the number of operations per cycle by performing the per-channel MAC operation as a partial summation by multiple processing elements.

[0058] 9 illustrates a more detailed method 110 of operating a performance-enhanced computing system. As previously described, the method 110 may be implemented generally in a hardware accelerator, such as, for example, the HWA 50 (FIG. 4). More specifically, the method 110 may be implemented in one or more modules, as a set of logic instructions stored in a machine or computer-readable storage medium, such as RAM, ROM, PROM, firmware, flash memory, or the like, configurable logic, such as, for example, PLA, FPGA, CPLD, fixed function logic hardware, such as, for example, using circuit technology, such as ASIC, CMOS, or TTL technology, or any combination thereof.

[0059] The illustrated processing block 112 selects a type of rulebook based on application information, where the type of rulebook is one of i2o type or o2i type. Block 114 may allocate memory based on the sparsity of the data. For example, block 114 may provide dynamic allocation of smaller chunks of memory based on the sparsity, which may help accommodate approximately 1.5 times more rulebook lines compared to storage mapped as a fixed resource per weight plane. In an embodiment, block 116 reads data from the L1 cache, where the data has variables with dimensions greater than 2 and the data is in a rulebook line format that is associated with the type of rulebook. Thus, block 116 enables the architecture to process weight planes hierarchically by selecting several planes at a time based on the functional block area budget. In one example, block 118 decodes the data with instructions indicating the number of rulebooks to be processed, the input feature size, the output feature size, and multiple feature map base addresses.

[0060] Furthermore, block 120 may rearrange the spatially distributed voxel OFM to the decoded data based on the weight plane. In an embodiment, block 122 swaps the output data from the index queue based on the type of the rulebook. Thus, block 122 allows both i2o type and o2i type rulebooks to be scheduled in the same scheduler architecture. Block 124 shown performs a per-channel MAC operation on the rearranged spatially distributed voxel OFM to obtain the output. Here, the per-channel MAC operation is performed as a partial addition by multiple processing elements. Accordingly, blocks 112, 114, 116 and 122 further enhance performance.

[0061] 10 illustrates a method 130 for managing feature storage. As previously described, the method 130 may be implemented generally in a hardware accelerator, such as, for example, the HWA 50 (FIG. 4). More specifically, the method 130 may be implemented in one or more modules, as a set of logic instructions stored in a machine or computer readable storage medium, such as RAM, ROM, PROM, firmware, flash memory, etc., configurable logic, such as, for example, PLA, FPGA, CPLD, fixed function logic hardware, such as, for example, ASIC, using circuit technology, such as CMOS or TTL technology, or any combination thereof.

[0062] The illustrated processing block 132 provides for storing the feature pair information (e.g., IFM-OFM) in a static buffer. In an embodiment, block 134 stores index information corresponding to the feature mapping pair information in a data storage location. The method 130 can significantly reduce the amount of storage used.

[0063] 11, a performance-enhanced computing system 140 is shown. System 140 may generally be part of an electronic device / platform having computing capabilities (e.g., personal digital assistant / PDA, notebook computer, tablet computer, convertible tablet, server), communication capabilities (e.g., smartphone), imaging capabilities (e.g., camera, camcorder), media playback capabilities (e.g., smart television / TV), wearable capabilities (e.g., watch, glasses, hat, footwear, jewelry), vehicle capabilities (e.g., car, truck, bike), robotic capabilities (e.g., autonomous robot), Internet of Things (IoT) capabilities, etc., or any combination thereof. In the illustrated example, system 140 includes a host processor 142 (e.g., central processing unit / CPU) having an integrated memory controller (IMC) 144 coupled to a system memory 146.

[0064] The illustrated system 140 also includes an input / output (IO) module 148 implemented along with the host processor 142, accelerator 151, and graphics processor 150 (e.g., graphics processing unit / GPU) on a semiconductor die 152 as a system on chip (SoC). The illustrated IO module 148 communicates with, for example, a display 154 (e.g., touch screen, liquid crystal display / LCD, light emitting diode / LED display), a network controller 156 (e.g., wired and / or wireless), and mass storage 158 (e.g., hard disk drive / HDD, optical disk, solid state drive / SSD, flash memory).

[0065] In an embodiment, as previously described, accelerator 151 includes logic 160 (e.g., logic instructions, configurable logic, fixed function hardware logic, etc., or any combination thereof) for performing one or more aspects of method 100 (FIG. 8), method 110 (FIG. 9), and / or method 130 (FIG. 10). Thus, logic 160 may decode data with instructions indicating a number of rulebooks to be processed, an input feature size, an output feature size, and a number of feature map base addresses. Logic 160 may also rearrange the output feature map of spatially distributed voxels based on the weight planes in the decoded data, and perform a per-channel MAC operation on the rearranged spatially distributed voxel output feature map to obtain an output. In an embodiment, the per-channel MAC operation is performed as a partial addition by a number of processing elements within accelerator 151, host processor 142, graphics processor 150, IO module 148, and / or semiconductor die 152.

[0066] Computing system 140 is considered performance enhanced at least to the extent that the instructions facilitate more seamless and efficient execution, reduce computational and memory requirements by rearranging spatially distributed voxels based on weight planes, and / or increase the number of operations per cycle by performing per-channel MAC operations as partial additions by multiple processing elements.

[0067] FIG. 12 illustrates a semiconductor packaged device 170. The illustrated device 170 includes one or more substrates 172 (e.g., silicon, sapphire, gallium arsenide) and logic 174 (e.g., transistor arrays and other integrated circuits / IC components) coupled to the substrate 172. The logic 174 may be implemented at least in part in configurable logic or fixed function logic hardware. As previously described, in one example, the logic 174 implements one or more aspects of the method 100 ( FIG. 8 ), the method 110 ( FIG. 9 ), and / or the method 130 ( FIG. 10 ). Thus, the logic 174 may decode data with instructions indicating the number of rule books to be processed, the input feature size, the output feature size, and a number of feature map base addresses. Logic 174 may also rearrange the spatially distributed voxel output feature map in the decoded data based on the weight plane and perform a per-channel MAC operation on the rearranged spatially distributed voxel output feature map to obtain an output. In an embodiment, the per-channel MAC operation is performed as a partial summation by multiple processing elements in logic 174 coupled to one or more substrates 172.

[0068] Apparatus 170 is considered performance enhanced at least to the extent that the instructions facilitate more seamless and efficient execution, reduce computation and memory requirements by rearranging spatially distributed voxels based on weight planes, and / or increase the number of operations per cycle by performing per-channel MAC operations as partial additions by multiple processing elements.

[0069] In one example, logic 174 includes transistor channel regions that are disposed (e.g., buried) within substrate 172. Thus, the interface between logic 174 and substrate 172 may not be an abrupt coupling. Logic 174 may also be considered to include an epitaxial layer grown on an initial wafer of substrate 172.

[0070] FIG. 13 illustrates a processor core 200, according to one embodiment. Processor core 200 may be a core for any type of processor, such as a microprocessor, embedded processor, digital signal processor (DSP), network processor, or other device that executes code. Although only one processor core 200 is shown in FIG. 13, a processing element may alternatively include more than one of the processor cores 200 illustrated in FIG. 13. Processor core 200 may be a single-threaded core, or, for at least one embodiment, processor core 200 may be multi-threaded in that it may include more than one hardware thread context (or "logical processor") per core.

[0071] FIG. 13 also illustrates memory 270 coupled to processor core 200. Memory 270 may be any of a wide variety of memories (including various layers of a memory hierarchy) known or otherwise available to one of skill in the art. Memory 270 may include one or more code 213 instructions executed by processor core 200, where code 213 may implement one or more aspects of method 100 (FIG. 8), method 110 (FIG. 9), and / or method 130 (FIG. 10), as previously described. Processor core 200 follows a program sequence of instructions indicated by code 213. Each instruction may enter front end section 210 and may be processed by one or more decoders 220. Decoder 220 may generate as its output micro-operations, such as fixed-width micro-operations of a predefined format, or may generate other instructions, micro-instructions, or control signals that reflect the original code instructions. The illustrated front end portion 210 also includes register renaming logic 225 and scheduling logic 230, which generally allocate resources and queue operations corresponding to the transformation instructions for execution.

[0072] Processor core 200 is shown to include execution logic 250 having a set of execution units 255-1 through 255-N. Some embodiments may include multiple execution units dedicated to a particular function or set of functions. Other embodiments may include only one execution unit or may include one execution unit capable of performing a particular function. The execution logic 250 is shown to perform operations specified by code instructions.

[0073] After completing execution of the operations specified by the code instructions, back-end logic 260 retires the instructions of code 213. In one embodiment, processor core 200 allows out-of-order execution but requires in-order retirement of instructions. Retirement logic 265 may take a variety of forms known to those skilled in the art (e.g., a reorder buffer or the like). In this manner, processor core 200 is transformed during execution of code 213 in terms of at least the outputs generated by the decoders, the hardware registers and tables utilized by register renaming logic 225, and any registers (not shown) modified by execution logic 250.

[0074] 13, a processing element may include other elements on a chip with processor core 200. For example, a processing element may include memory control logic along with processor core 200. A processing element may include I / O control logic and / or I / O control logic integrated with the memory control logic. A processing element may also include one or more caches.

[0075] 14, there is shown a block diagram of an embodiment of a computing system 1000 in accordance with an embodiment. Shown in FIG. 14 is a multiprocessor system 1000 including a first processing element 1070 and a second processing element 1080. Although two processing elements 1070 and 1080 are shown, it will be understood that an embodiment of the system 1000 may also include only one such processing element.

[0076] System 1000 is shown as a point-to-point interconnect system, in which a first processing element 1070 and a second processing element 1080 are coupled via a point-to-point interconnect 1050. It should be understood that any or all of the interconnects shown in Figure 14 may be implemented as multi-drop buses rather than point-to-point interconnects.

[0077] As shown in Figure 14, each of the processing elements 1070 and 1080 may be a multi-core processor including a first processor core and a second processor core (i.e., processor cores 1074a and 1074b, and processor cores 1084a and 1084b). Such cores 1074a, 1074b, 1084a, 1084b may be configured to execute instruction codes in a manner similar to that described above in connection with Figure 13.

[0078] Each processing element 1070, 1080 may include at least one shared cache 1896a, 1896b. The shared cache 1896a, 1896b may store data (e.g., instructions) utilized by one or more components of the processor, such as the cores 1074a, 1074b and 1084a, 1084b, respectively. For example, the shared cache 1896a, 1896b may locally cache data stored in the memory 1032, 1034 for faster access by the components of the processor. In one or more embodiments, the shared cache 1896a, 1896b may include one or more mid-level caches, such as a level 2 (L2), level 3 (L3), level 4 (L4) or other level cache, a last level cache (LLC), and / or combinations thereof.

[0079] While only two processing elements 1070, 1080 are shown, it should be understood that the scope of the embodiments is not so limited. In other embodiments, one or more additional processing elements may be present in a given processor. Alternatively, one or more of the processing elements 1070, 1080 may be elements other than processors, such as accelerators or field programmable gate arrays. For example, the additional processing elements may include additional processors that are identical to the first processor 1070, additional processors that are heterogeneous or asymmetric to the first processor 1070, accelerators (e.g., graphics accelerators or digital signal processing (DSP) units, etc.), field programmable gate arrays, or any other processing elements. Various differences may exist between the processing elements 1070, 1080 in terms of a wide variety of value criteria, including architectural characteristics, microarchitectural characteristics, thermal characteristics, power consumption characteristics, and the like. These differences may effectively manifest themselves as asymmetries and heterogeneities between the processing elements 1070, 1080. For at least one embodiment, the various processing elements 1070, 1080 may be in the same die package.

[0080] The first processing element 1070 may further include a memory controller logic (MC) 1072 and point-to-point (PP) interfaces 1076 and 1078. Similarly, the second processing element 1080 may include a MC 1082 and PP interfaces 1086 and 1088. As shown in FIG. 14, the MCs 1072 and 1082 couple the processors to respective memories, i.e., memory 1032 and memory 1034. These memories may be part of a main memory locally attached to the respective processors. Although the MCs 1072 and 1082 are shown as being integrated into the processing elements 1070, 1080, for alternative embodiments, the MC logic may be separate logic external to the processing elements 1070, 1080 rather than being integrated therewith.

[0081] The first processing element 1070 and the second processing element 1080 may be coupled to the I / O subsystem 1090 via PP interconnects 1076, 1086, respectively. As shown in FIG. 14, the I / O subsystem 1090 includes PP interfaces 1094 and 1098. Additionally, the I / O subsystem 1090 includes an interface 1092 that couples the I / O subsystem 1090 with a high-performance graphics engine 1038. In one embodiment, the bus 1049 may be used to couple the graphics engine 1038 to the I / O subsystem 1090. Alternatively, a point-to-point interconnect may couple these components.

[0082] I / O subsystem 1090 may in turn be coupled to a first bus 1016 via an interface 1096. In one embodiment, first bus 1016 may be a bus such as a Peripheral Component Interconnect (PCI) bus, or a PCI Express bus, or other third generation I / O interconnect bus, although embodiments are not limited in scope thereto.

[0083] As shown in FIG. 14, various I / O devices 1014 (e.g., biometric scanner, speaker, camera, sensors) may be coupled to a first bus 1016 along with a bus bridge 1018 that may couple the first bus 1016 to a second bus 1020. In one embodiment, the second bus 1020 may be a low pin count (LPC) bus. In one embodiment, various devices may be coupled to the second bus 1020 including, for example, a keyboard / mouse 1012, a communication device 1026, and a data storage unit 1019 such as a disk drive or other mass storage device that may include code 1030. As previously described, the illustrated code 1030 may implement one or more aspects of method 100 (FIG. 8), method 110 (FIG. 9), and / or method 130 (FIG. 10). Additionally, audio I / O 1024 may be coupled to a second bus 1020 , and a battery 1010 may provide power to the computing system 1000 .

[0084] It should be noted that other embodiments are contemplated. For example, rather than the point-to-point architecture of Figure 14, the system could implement a multi-drop bus or other such communication topology. Also, the elements of Figure 14 could alternatively be split using more or fewer integrated chips than those shown in Figure 14.

[0085] [Additional notes and examples]

[0086] (Example 1) A performance-enhanced computing system comprising: a network controller; a processor coupled to the network controller; and a memory coupled to the processor, the memory including a set of executable program instructions that, when executed by the processor, cause the computing system to: decode data using instructions indicating a number of rule books to be processed, an input feature size, an output feature size, and a plurality of feature map base addresses; rearrange an output feature map of spatially distributed voxels based on a weight plane to the decoded data; and perform a per-channel multiply-accumulate (MAC) operation on the rearranged spatially distributed voxel output feature map to obtain an output, the per-channel MAC operation being performed as a partial addition by a plurality of processing elements within the processor.

[0087] (Example 2) The set of executable program instructions, when executed, further causes the computing system to select a type of rulebook based on application information, the type of rulebook being one of an input-to-output type or an output-to-input type, including the computing system of Example 1.

[0088] (Example 3) The computing system of Example 2 includes the set of executable program instructions, when executed, further causing the computing system to read the data from a level 1 (L1) cache, the data having a variable with a dimension greater than two, and the data being in a rulebook line format associated with the rulebook type.

[0089] (Example 4) The computing system of Example 2, wherein the set of executable program instructions, when executed, further causes the computing system to swap output data from index queues based on the type of the rule book.

[0090] (Example 5) The computing system of Example 1, wherein the set of executable program instructions, when executed, further causes the computing system to allocate memory and the plurality of processing elements based on sparsity of the data, and the per-channel MAC operation identifies duplicate output feature maps and locally merges the duplicate output feature maps.

[0091] (Example 6) The computing system of any one of Examples 1 to 5, further including a static buffer and a data storage, wherein the set of executable program instructions, when executed, further causes the computing system to store feature mapping pair information in the static buffer and store index information corresponding to the feature mapping pair information in a location within the data storage.

[0092] (Example 7) A semiconductor device includes one or more substrates and logic coupled to the one or more substrates, the logic being at least partially implemented in one or more configurable logic or fixed-function hardware logic, the logic coupled to the one or more substrates performing the following operations: decoding data using instructions indicating a number of rule books to be processed, an input feature size, an output feature size, and a plurality of feature map base addresses; rearranging a spatially distributed voxel output feature map based on a weight plane to the decoded data; and performing a per-channel multiply-accumulate (MAC) operation on the rearranged spatially distributed voxel output feature map to obtain an output, the per-channel MAC operation being performed as a partial addition by a plurality of processing elements in the logic coupled to the one or more substrates.

[0093] (Example 8) The device of Example 7 includes the logic coupled to the one or more substrates, and the logic selects a type of rulebook based on application information, the type of rulebook being one of an input-to-output type or an output-to-input type.

[0094] (Example 9) The apparatus of Example 8 includes the logic coupled to the one or more substrates, reading the data from a level 1 (L1) cache, the data having variables with a dimension greater than two, and the data being in a rulebook line format associated with the rulebook type.

[0095] Example 10. The logic coupled to the one or more boards includes the apparatus of example 8, further configured to switch output data from index queues based on the type of rule book.

[0096] (Example 11) The apparatus of Example 7 includes, wherein the logic coupled to the one or more substrates allocates memory and the processing elements based on sparsity of the data, and the per-channel MAC operation identifies duplicate output feature maps and locally merges the duplicate output feature maps.

[0097] (Example 12) The logic coupled to the one or more substrates includes any one of the devices of Examples 7 to 11, which stores feature mapping pair information in a static buffer and stores index information corresponding to the feature mapping pair information in a data storage location.

[0098] (Example 13) The logic coupled to the one or more substrates includes any one of the devices of Examples 7 to 11, including a transistor channel region located within the one or more substrates.

[0099] (Example 14) includes at least one computer-readable storage medium comprising a set of executable program instructions that, when executed by a computing system, cause the computing system to decode data using instructions indicating a number of rule books to be processed, an input feature size, an output feature size, and a plurality of feature map base addresses; rearrange an output feature map of spatially distributed voxels on the decoded data based on a weight plane; and perform a per-channel multiply-accumulate (MAC) operation on the rearranged spatially distributed voxel output feature map to obtain an output, wherein the per-channel MAC operation is performed as a partial addition by a plurality of processing elements within the computing system.

[0100] (Example 15) The at least one computer-readable storage medium of Example 14 includes at least one computer-readable storage medium, wherein the set of executable program instructions, when executed, further causes the computing system to select a type of rulebook based on application information, the type of rulebook being one of an input-to-output type or an output-to-input type.

[0101] (Example 16) The at least one computer-readable storage medium of Example 15 includes at least one computer-readable storage medium, wherein the set of executable program instructions, when executed, further causes the computing system to read the data from a level 1 (L1) cache, the data having a variable with a dimension greater than two, and the data being in a rulebook line format associated with the rulebook type.

[0102] (Example 17) The set of executable program instructions includes at least one computer-readable storage medium of Example 15, which, when executed, further causes the computing system to swap output data from index queues based on the type of the rule book.

[0103] (Example 18) The at least one computer-readable storage medium of Example 14 includes at least one computer-readable storage medium, wherein the set of executable program instructions, when executed, further causes the computing system to allocate memory and the plurality of processing elements based on sparsity of the data, and the per-channel MAC operation identifies duplicate output feature maps and locally merges the duplicate output feature maps.

[0104] (Example 19) The set of executable program instructions includes at least one computer-readable storage medium of any one of Examples 14 to 18, which, when executed, further causes the computing system to store feature mapping pair information in a static buffer and store index information corresponding to the feature mapping pair information in a data storage location.

[0105] (Example 20) A method of operating a performance enhancing computing system, comprising: decoding data using instructions indicating a number of rule books to be processed, an input feature size, an output feature size, and a plurality of feature map base addresses; rearranging an output feature map of spatially distributed voxels into the decoded data based on a weight plane; and performing a per-channel multiply-accumulate (MAC) operation on the rearranged spatially distributed voxel output feature map to obtain an output, wherein the per-channel MAC operation is performed as a partial addition by a plurality of processing elements.

[0106] (Example 21) The method of Example 20, further comprising a step of selecting a type of rulebook based on application information, the type of rulebook being one of an input-to-output type or an output-to-input type.

[0107] (Example 22) The method of Example 21 further comprising reading the data from a level 1 (L1) cache, the data having variables with a dimension greater than two, and the data being in a rulebook line format associated with the rulebook type.

[0108] (Example 23) The method of Example 21, further comprising swapping output data from the index queue based on the type of the rulebook.

[0109] (Example 24) The method of Example 20 further includes allocating memory and the plurality of processing elements based on sparsity of the data, and the MAC operation for each channel includes identifying duplicate output feature maps and locally merging the duplicate output feature maps.

[0110] (Example 25) The method of any one of Examples 20 to 24, further comprising: storing feature mapping pair information in a static buffer; and storing index information corresponding to the feature mapping pair information in a data storage location.

[0111] (Example 26) A means for carrying out the method of any one of Examples 20 to 25 is included.

[0112] The embodiments are applicable for use with all types of semiconductor integrated circuit ("IC") chips. Examples of these IC chips include, but are not limited to, processors, controllers, chipset components, programmable logic arrays (PLAs), memory chips, network chips, systems on chips (SoCs), SSD / NAND controller ASICs, and the like. Additionally, in some figures, signal conductors are represented as lines, some of which may be different to indicate more component signal paths, may have number labels to indicate multiple component signal paths, and / or may have arrows on one or more ends to indicate the primary information flow direction. However, this should not be construed as limiting. Rather, such additional details may be used in connection with one or more exemplary embodiments to facilitate easier understanding of the circuit. Any signal lines represented, whether or not they have additional information, may in fact comprise one or more signals that may propagate in multiple directions and may be implemented with any suitable type of signaling, e.g., digital or analog lines implemented in differential pairs, fiber optic lines, and / or single-ended lines.

[0113] Although example sizes / models / values / ranges may be given, the embodiments are not limited to the same. It is expected that as manufacturing technologies (e.g., photolithography) mature over time, smaller devices may be manufactured. Furthermore, for simplicity of illustration and explanation, and so as not to obscure certain aspects of the embodiments, well-known power / ground connections to IC chips and other components may or may not be shown in the drawings. Furthermore, in view of the fact that configurations may be shown in block diagram form to avoid obscuring the embodiments, and details regarding the implementation of such block diagram configurations are highly dependent on the computing system in which the embodiments are implemented, i.e., such details should be within the knowledge of one skilled in the art. Where specific details (e.g., circuits) are described to explain exemplary embodiments, it should be apparent to one skilled in the art that the embodiments may be practiced without these specific details or with variations of these specific details. Thus, the description should be considered as illustrative and not limiting.

[0114] The term "coupled" may be used herein to refer to any type of direct or indirect relationship between the components under consideration and may apply to electrical, mechanical, fluid, optical, electromagnetic, electromechanical, or other connections. Additionally, terms such as "first," "second," and the like may be used herein for ease of description only and do not imply any particular temporal or chronological connotation unless otherwise stated.

[0115] As used in this application and the claims, a list of items joined by the term "one or more of" may mean any combination of the listed terms. For example, the phrase "one or more of A, B, or C" may mean A, B, C, A and B, A and C, B and C, or A, B, and C.

[0116] Those skilled in the art will appreciate from the foregoing description that the broad technology of the embodiments can be implemented in a variety of forms. Thus, although the embodiments have been described with reference to specific examples thereof, the true scope of the embodiments should not be so limited, as other modifications will become apparent to those skilled in the art upon review of the drawings, the specification, and the following claims.

Claims

1. Memory, a microcontroller circuit; The microcontroller circuit includes: rearrange activation values ​​based on a sparsity format, the rearranged activation values ​​being stored in an activation buffer; assigning first weights from a weight buffer based on a bit mask encoding information in the sparsity format and a first feature from the activation buffer to a first multiply-accumulate (MAC) circuit in a systolic array of MAC circuits, the first MAC circuit performing a first MAC operation to generate a first partial result; assigning second weights from the weight buffer based on the bit mask and second rearranged activation values ​​from the activation buffer to a second multiply-accumulate (MAC) circuit in the systolic array of a MAC circuit, the second MAC circuit performing a second MAC operation to generate a second partial result; adding the first partial result and the second partial result; A convolutional neural network (CNN) hardware accelerator that performs

2. the microcontroller circuit assigns the weights and the rearranged activation values ​​to the systolic array of a MAC circuit; 2. The CNN hardware accelerator of claim 1, wherein the weights and the rearranged activation values ​​have a first group processed at a first time and a second group processed at a second time.

3. 3. The CNN hardware accelerator of claim 2, wherein the first group and the second group have different numbers of non-zero elements.

4. 2. The CNN hardware accelerator of claim 1, wherein the systolic array of a MAC circuit shares values ​​in the weight buffer and the activation buffer to reduce the number of memory accesses.

Citation Information

Patent Citations

  • Clustering device, clustering method, program, and data structure

    JP2019204436A

  • Convolution arithmetic processing device and convolution arithmetic processing method

    JP2020140507A

  • Systems and methods for data management

    JP2020513637A

  • Neural network processing system having multiple processors and a neural network accelerator

    JP2020537786A

  • Sparse Optimization for Matrix Accelerator Architectures

    JP2022523762A