Distributed computing in a package with dram dies and a logic die

Abstract

An apparatus according to one embodiment of the present disclosure comprises a first compute component, a second compute component, a first Through Silicon Via (TSV) set associated with the first compute component and connecting stacked dies of the apparatus, a first die comprising a second TSV set associated with the second compute component and connecting stacked dies of the apparatus, and a second die stacked on the first die, wherein the second die may include a first memory bank module connected to the first compute component using the first TSV set and a second memory bank module connected to the second compute component using the second TSV set.

Description

Technology Field

[0001] Some embodiments of the present disclosure generally relate to semiconductor circuits. More specifically, according to the present disclosure, it relates to a semiconductor microchip package implementing a distributed computing system comprising a memory die and a logic die designed for artificial intelligence (AI) operations and / or applications. Background Technology

[0002] High Bandwidth Memory (HBM) is a high-performance memory system composed of multiple 3D stacked Dynamic Random Access Memory (DRAM) dies. Various applications, such as deep neural networks and artificial intelligence (AI), may require massive computational and memory capabilities to train on different datasets and learn with high accuracy. For these applications, high memory bandwidth can be desirable. Memory bandwidth can be described by core bandwidth and bus bandwidth. As the number of stacked DRAM dies sharing the same bus increases, bus bandwidth can become a limiting factor in memory performance.

[0003] The continuous expansion of AI demands increasingly higher memory bandwidth and capacity. Attempts to expand memory bandwidth optimized for AI-based applications using stacked HBM architectures (e.g., increasing memory bandwidth for larger neural networks and / or more complex AI computations) can lead to various drawbacks. For instance, these may include shoreline constraints (e.g., limited pin counts) and high power consumption associated with using physically large stacked HBMs to expand memory bandwidth. Additionally, the movement of large volumes of high-bandwidth data between stacked DRAM dies and on-chip processing elements (e.g., logic dies, hosts, etc.) can lead to increased energy consumption. Furthermore, while architectures that implement logic on HBM core dies can provide computing power, they carry the risk of exceeding thermal budget limits associated with the die and / or package.

[0004] Therefore, there is a need for a flexible system architecture that performs computations by utilizing a standalone distributed computing system that is cop-packaged (cell on peri) beneath the DRAM die to reduce or minimize data traffic (e.g., reduced power consumption, elimination of the physical global bus) and maintain the thermal budget, while providing a high-bandwidth, energy-efficient memory subsystem that can be optimized for AI-based applications.

[0005] The information disclosed above in this background section is intended to enhance understanding of the context of the present invention and, accordingly, may include information that does not constitute prior art. The problem to be solved

[0006] The problem that the technical concept of the present disclosure aims to solve is to provide distributed computing within a package including a DRAM die and a logic die. means of solving the problem

[0007] Aspects of some embodiments of this disclosure generally relate to Compute with Memory Stack (CMS) semiconductor hardware (e.g., chips, packages, etc.) having an architecture that integrates memory and computing into a single chip, enabling high-bandwidth communication over shorter distances between elements and minimizing data traffic. In some embodiments, the device comprises a first compute component, a second compute component, a first Through Silicon Via (TSV) set associated with the first compute component and connecting stacked dies of the device, a first die comprising a second TSV set associated with the second compute component and connecting stacked dies of the device, and a second die stacked on the first die, wherein the second die may include a first memory bank module connected to the first compute component using the first TSV set and a second memory bank module connected to the second compute component using the second TSV set.

[0008] In some embodiments, the device further comprises a third die stacked on the second die, and the third die may include a third memory bank module connected to the first compute component using the first TSV set and a fourth memory bank module connected to the second compute component using the second TSV set.

[0009] In some embodiments, the first memory bank module connected to the first compute component using the first TSV set forms a first processing element configured to perform compute functions using data from the first memory bank module, and the second memory bank module connected to the second compute component using the second TSV set forms a second processing element configured to perform compute functions using data from the second memory bank module.

[0010] In some embodiments, the device further comprises a plurality of interconnected ports of the first die, the ports connecting the first processing element and the second processing element to one or more additional processing elements, the first processing element and the second processing element comprising a compute circuit configured to perform a compute function, the one or more additional processing elements comprising a compute component connected to at least one memory bank module, and the at least one memory bank module may be selected from an array of memory bank modules arranged on the first die and stacked on the first die using a corresponding set of TSVs.

[0011] In some embodiments, for the one or more additional processing elements, each compute component may be configured to perform a computation function on data from each of at least one memory bank module stacked thereon.

[0012] In some embodiments, the device may further include a controller programmed to execute the transfer of data between the first compute component and the first memory bank module, the transfer of data between the second compute component and the second memory bank module, and the transfer of data between each of at least one memory bank module and each of the compute components for one or more additional processing elements.

[0013] In some embodiments, the first TSV set and the second TSV set may be some of a plurality of TSVs distributed in a plurality of regions of the first die.

[0014] In some embodiments, the first TSV set may include at least one of the plurality of TSVs located in a first region of the first die in contact with the first compute component, and the second TSV set may include at least one of the plurality of TSVs located in a second region of the first die in contact with the second compute component.

[0015] In some embodiments, the first processing component may further include the third memory bank module and be configured to perform compute functions using data from the third memory bank module, and the second processing component may further include the fourth memory bank module and be configured to perform compute functions using data from the fourth memory bank module.

[0016] In some embodiments, the stacked dies of the device may include a plurality of additional dies stacked on the third die.

[0017] In some embodiments, the device further includes dynamic random access memory (DRAM) dies, and one or more of the second die or the third die may include a DRAM die.

[0018] In some embodiments, the compute function may include one or more of matrix multiplication, inner product, activation function, or mathematical function related to an AI (Artificial Intelligence) application.

[0019] In some embodiments, the device comprises a first die and a second die stacked on the first die, wherein the first die comprises a plurality of TSVs distributed in a plurality of regions of the first die, a first compute component, a second compute component, a first set of TSVs associated with the first compute component and connecting the stacked dies of the device, and a second set of TSVs associated with the second compute component and connecting the stacked dies of the device, wherein the first set of TSVs comprises one or more of the plurality of TSVs within a first region of the first die, and the second set of TSVs comprises one or more of the plurality of TSVs within a second region different from the first region of the first die, and the second die may comprise a first memory bank module connected to the first compute component using the first set of TSVs and a second memory bank module connected to the second compute component using the second set of TSVs.

[0020] In some embodiments, the first memory bank module connected to the first compute component using the first TSV set forms a first processing element configured to perform compute functions using data from the first memory bank module, and the second memory bank module connected to the second compute component using the second TSV set forms a second processing element configured to perform compute functions using data from the second memory bank module.

[0021] In some embodiments, the stacked dies of the device may include a plurality of additional dies stacked on the second die.

[0022] In some embodiments, the device further includes a dynamic random access memory (DRAM) die, and the second die or one or more of the plurality of dies may include a DRAM die.

[0023] In some embodiments, the plurality of TSVs may be configured to transfer data stored in the DRAM dies to the first die.

[0024] In some embodiments, the device may further include a plurality of interconnect ports on the first die configured to connect the first processing element and the second processing element to a plurality of processing elements in order to implement a distributed computing system.

[0025] In some embodiments, the device may further include a controller programmed to execute data transfer between the first compute component and the first memory bank module and data transfer between the second compute component and the second memory bank module.

[0026] In some embodiments, the system includes an accelerator processor configured to perform a compute function and a memory that stores instructions for performing the compute function when executed by the accelerator processor, and the accelerator processor includes a first compute component, a second compute component, a first Through Silicon Via (TSV) set associated with the first compute component and connecting stacked dies of the device, and a first die including a second TSV set associated with the second compute component and connecting stacked dies of the device, and a second die stacked on the first die, and the second die may include a first memory bank module connected to the first compute component using the first TSV set and a second memory bank module connected to the second compute component using the second TSV set. Brief explanation of the drawing

[0027] The following section describes an overview of the present invention with reference to exemplary embodiments shown in the drawings. FIG. 1a is a perspective view of an exemplary Compute with Memory Stack (CMS) device implementing a distributed computing system of processing elements shown in FIG. 1b on a die, according to some embodiments of the present disclosure. FIG. 1b illustrates an example of a processing element having a microarchitecture included in the CMS device of FIG. 1a, according to some embodiments of the present disclosure. FIG. 1c is a side view of the CMS device shown in FIG. 1a according to some embodiments of the present disclosure. FIG. 2a is a plan view of an exemplary CMS device shown in FIG. 1a, according to some embodiments of the present disclosure. FIG. 2b is a plan view of another exemplary CMS device according to some embodiments of the present disclosure. FIG. 3 illustrates another exemplary CMS device having an architecture including a 4-hi stack of memory bank modules according to some embodiments of the present disclosure. FIG. 4 illustrates another exemplary CMS device having an architecture including an 8-layer stack of memory bank modules according to some embodiments of the present disclosure. FIG. 5 illustrates another exemplary CMS device having a multi-die package configuration including a plurality of CMS devices illustrated in FIG. 3, according to some embodiments of the present disclosure. FIG. 6 is a flowchart illustrating an exemplary operation of a method for transmitting data within the architecture of the CMS device of FIG. 1a, according to some embodiments of the present disclosure. FIG. 7 is a block diagram of an exemplary electronic device for implementing the hardware of the CMS device shown in FIG. 1a, according to some embodiments of the present disclosure. Specific details for implementing the invention

[0028] In the following detailed description, numerous specific details are described to provide a complete understanding of the present disclosure. However, it will be understood by those skilled in the art that aspects disclosed may be practiced without these specific details. In other cases, well-known methods, procedures, components, and circuits have not been described in detail to avoid obscuring the subject matter disclosed herein.

[0029] Throughout this specification, references to "one embodiment" or "an embodiment" mean that specific features, structures, or characteristics described in relation to an embodiment may be included in at least one embodiment disclosed herein. Accordingly, the appearance of phrases such as "in one embodiment," "in an embodiment," "according to one embodiment," or other phrases with a similar meaning in various places throughout this specification may not all refer to the same embodiment. Furthermore, specific features, structures, or characteristics may be combined in any appropriate manner in one or more embodiments. In this regard, the word "exemplary" as used herein means "serving as an example, case, or illustration." An embodiment described as "exemplary" in this specification is not to be interpreted as being preferred or advantageous over other embodiments. Furthermore, specific features, structures, or characteristics may be combined in any appropriate manner in one or more embodiments. Additionally, depending on the context of the discussion in this specification, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. Likewise, hyphenated terms (e.g., "two-dimensional", "pre-determined", "pixel-specific", etc.) may sometimes be used interchangeably with their non-hyphenated versions (e.g., "two dimensional", "predetermined", "pixel specific", etc.). Items written in uppercase (e.g., "Counter Clock", "Row Select", "PIXOUT", etc.) may sometimes be used interchangeably with their corresponding non-uppercase versions (e.g., "counter clock", "row select", "pixout", etc.). These sometimes interchangeable uses are not considered to be inconsistent with each other.

[0030] Additionally, depending on the context of the discussion in this document, singular terms may include their corresponding plural forms, and plural terms may include their corresponding singular forms. Furthermore, the various figures (including configuration diagrams) shown and discussed in this document are for illustrative purposes only and are not drawn to scale. For example, the dimensions of some elements may be exaggerated relative to others for clarity. Also, reference numbers are repeated between figures to indicate relevant and / or similar elements where deemed appropriate.

[0031] The terms used herein are for the purpose of describing some embodiments only and are not intended to limit the scope of the claims. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless otherwise indicated in the context. It will be further understood that when used herein, the terms “comprising” and / or “comprising” specify the presence of the specified features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0032] If an element or layer is referred to as being on top of, "connected to," or "combined" with another element or layer, it will be understood that it is directly on top of, connected to, or combined with the other element or layer, or that an intermediate element or layer may exist. Conversely, if an element is referred to as being "directly," "directly connected," or "directly combined" with another element or layer, no intermediate element or layer exists. The same (similar) (like) symbol refers to the same (similar) element throughout. The term "and / or" as used herein includes any combination of one or more related items among the listed items.

[0033] Terms such as 'first' and 'second' used herein serve as labels for the preceding nouns and do not imply any type of order (e.g., spatial, temporal, logical, etc.) unless explicitly defined. Additionally, the same reference numerals may be used in two or more figures to refer to parts, components, blocks, circuits, units, or modules having the same or similar functions. However, such use is intended for the simplification of examples and for ease of discussion, and does not imply that the configuration or architectural details of such components or units are identical in all embodiments, or that such common reference parts or modules are the only way to implement some of the embodiments disclosed herein.

[0034] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as generally understood by a person skilled in the art to which this subject matter belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with that in the context of the relevant technology, and will be further understood that they should not be interpreted in an idealized or overly formal sense unless explicitly defined otherwise herein.

[0035] As used in this document, "module" refers to any combination of software, firmware, and / or hardware configured to provide the functions described in this document in relation to the module. For example, software may be implemented as software packages, code and / or instruction sets or instructions, and the term "hardware" as used in the implementations described in this document may include, for example, assemblies, wired circuits, programmable circuits, state machine circuits, and / or firmware storing instructions executed by programmable circuits, either alone or in any combination. Modules may be implemented as circuits that constitute part of a larger system, collectively or individually, and include, but are not limited to, integrated circuits (ICs), systems on chips (SoCs), assemblies, etc.

[0036] In the field of computer technology, AI and other deep learning applications are becoming increasingly common, and current demand is very high. With the growth of AI computing applications, new hardware may be required to enable new applications in various areas, such as image and voice classification, media analysis, healthcare, autonomous machines, and smart assistants. For example, in deep neural network algorithms, the size of datasets can exceed the available computational power of the hardware. Furthermore, as emerging AI applications become more expansive and complex, massive computational and memory performance may be required to train on diverse datasets and learn with high accuracy. In addition, as applications such as high-performance computers (HPC) and graphics algorithms become data and computationally intensive, energy efficiency and low latency may become increasingly important.

[0037] Some server computing environments can utilize configurations that allow some computations to be performed physically closer to where data is stored. By implementing data movement over relatively short distances, higher bandwidth and reduced energy consumption can be achieved for computing systems performing highly complex calculations, such as AI applications. An example of a hardware architecture that can be modified to support computation (or logic) near memory is High Bandwidth Memory (HBM). HBM is a type of computer memory designed to achieve high-speed data transfer through a stacked architecture of memory chips. For instance, HBM is a high-performance memory implemented in a smaller form factor and consuming less power while achieving higher bandwidth by stacking Dynamic Random Access Memory (DRAM) dies on top of each other.

[0038] HBM2, an evolution of HBM, can contain up to 12 dies per stack, for example, and provides fin transfer speeds of 2.4 GT / s (gigabits per second) or higher. HBM and HBM2 have been used in hardware implementations, such as parallel accelerators, to provide the high-bandwidth, large capacity, and computational power required for AI applications. Subsequent versions of HBM (e.g., HBM3, HBM3e, HBM4, etc.) can provide higher memory capacity, greater transfer speeds, and more stacked DRAM dies. For brevity, all versions of HBM are referred to simply as "HBM" in this document.

[0039] HBM provides large-capacity memory through stacked memory chips, but computations within the architecture are performed on a host or other external chips. The host can include, for example, a CPU (central processing unit) such as a microprocessor, an ASIC (application-specific integrated circuit), a GPU (graphics processing unit), or a FPGA (field-programmable gate array). Therefore, HBM utilizes an asynchronous communication interface with the host. For example, in an HBM architecture, multiple stacked DRAM dies can be configured to share a global physical bus to support data movement between memory elements to computing elements such as the host (e.g., memory-utilization components).

[0040] In higher HBM stacking configurations (e.g., 8-stack DRAM dies, 12-stack DRAM dies, etc.), the utilization of memory bus bandwidth can become increasingly critical. Complex functions such as AI and HPC may require significant amounts of computational power and memory bandwidth, which further heightens the importance of bus bandwidth utilization in HBM architectures for these applications. As demands for memory bandwidth increase due to factors such as increasingly powerful GPUs, HBM architectures may not be able to fully utilize bus bandwidth. Additionally, while the asynchronous nature of communication over the global bus can improve performance, it makes processing complex logic operations more difficult. Furthermore, since HBM architectures require data movement between stacked DRAM dies and a host (e.g., compute), transmitting high-bandwidth data over relatively long distances can increase power consumption in HBM systems.

[0041] To address these problems and other limitations associated with some memory architectures, embodiments disclosed in the present invention provide Compute with Memory Stack (CMS) semiconductor hardware (e.g., die, package, etc.) having a distinct architecture that integrates memory and computing on a single die, enables high-bandwidth communication over shorter distances between elements, and minimizes data traffic. Accordingly, the CMS hardware disclosed in accordance with the present disclosure can provide memory devices with increasingly higher bandwidth, increased capacity, and high energy efficiency, optimized for complex processing functions such as AI applications.

[0042] The disclosed CMS hardware architecture may implement a die comprising a memory layer configured to support high-bandwidth memory and a compute layer configured to implement logic and / or computing functions disposed adjacent to (e.g., beneath) the memory layer. The memory layer and the compute layer may be interconnected (e.g., physically and / or telecommunicationally) using vias (e.g., Through Silicon Via (TSV)) distributed across the die, thereby supporting computing functions and / or performance implemented near memory on the die and eliminating the use of a global physical bus (e.g., required in HBM architecture). In one or more embodiments, the memory layer of the CMS architecture may include one or more stacked "layers" of memory bank modules. In one or more embodiments, the stacked memory bank modules in the memory layer of the CMS hardware architecture may be implemented as a stacked DRAM die. Accordingly, the CMS hardware architecture disclosed in accordance with the present disclosure can implement an integrated high-bandwidth memory and active computing device rather than a passive memory device (e.g., one that does not require a separate host for processing, as in existing HBM architectures). As disclosed in the present invention, the disclosed CMS hardware can realize various advantages, such as improving the speed and performance of data processing tasks by eliminating data transmission delays (e.g., delays related to communication with a separate computing SoC, global bus, etc.), and increasing the overall efficiency of hardware utilized in complex AI applications by enabling real-time analysis and faster decision-making.

[0043] In one or more embodiments, the CMS hardware architecture may be implemented using various configurations. For example, the CMS die may be configured to include a variable number of stacked memory bank modules. Additionally, in one or more embodiments, the CMS hardware architecture may provide a semiconductor package that includes a variable number of interconnected CMS dies in a manner that supports a wide range of bandwidth and / or capacity performance for memory. Accordingly, the embodiments disclosed in the present invention provide a flexible and / or adaptable CMS hardware architecture that allows memory and active computing devices to be scaled and / or optimized as desired to adequately support specific AI applications.

[0044] Additionally, embodiments disclosed in the present invention may provide processing elements that can serve as "building blocks" for a larger architecture of CMS hardware. For example, a CMS die may include an array of processing elements (e.g., multiple processing elements interconnected on the die via interconnection ports), and the size and / or dimensions of said array may be flexibly adjusted to be scaled and / or optimized based on applicable AI capabilities. In one or more embodiments, the processing element may have a microarchitecture including at least one memory bank module and may include computational logic positioned below the memory bank module and having a form factor substantially similar to said memory bank module, thereby supporting processing capabilities close to memory (e.g., memory bank module) while minimizing data traffic and operating within an appropriate thermal budget. The microarchitecture for said processing element is disclosed in relation to said framework of said CMS.

[0045] FIG. 1a is an exemplary perspective view of a CMS device (100) disclosed herein. In the example of FIG. 1A, the architecture of the CMS device (100) implements integrated high-bandwidth memory and computing functions and / or capabilities on the hardware of a semiconductor die (105). FIG. 1c is a side view of the CMS device (100) shown in FIG. 1a according to some embodiment of the present invention. As used herein, the term “die” may refer to a semiconductor device comprising functional circuits and / or integrated circuits (ICs) for performing functions that can be produced (e.g., monolithically) on a wafer of a semiconductor material such as silicon. In the architecture of the CMS device (100), the die (105) may be placed on the lower layer (e.g., bottom) of the architecture and may be composed of a “base die” or “first die” that implements the compute layer (110) (and compute components within it). As the base die, the die (105) may have one or more additional dies placed adjacently thereon (e.g., a memory module (116) implemented with memory dies stacked on top of the base die (105)). In one or more embodiments, the architecture of the CMS hardware may be described as being composed of an N x N (or NXM) array of smaller processing elements (150). The microarchitecture of the processing elements (150) is described in more detail with reference to FIG. 1b. As illustrated in FIG. 1a, an exemplary CMS device (100) may be composed of a 4 x 4 array of processing elements (150) implemented using a base die (105) (other array dimensions may also be used).FIG. 1a shows that an exemplary architecture for a CMS device (100) may include two layers (indicated by horizontal dashed lines), and the two layers may consist of a compute layer (110) and a memory layer (115) implemented on a die (105) (e.g., the die (105) is a base die including other components (compute circuit (151) of the computer layer (110)), the memory layer (115) is physically adjacent (e.g., upper) to the compute layer (110) and connected to the compute layer (110) using a number of vias (indicated by TSVs (152)) distributed across the die (105), and also external components (121, 122). Accordingly, each processing element (150) in the array may include a memory layer (115) (including a memory module (116)), a compute layer (110) (including compute components corresponding to a specific memory module (116) stacked thereon), and an associated TSV (152) arranged to provide memory and compute components to each processing element (150) of the array (connecting the compute components to the corresponding memory module stacked thereon). That is, the compute layer (110) implemented on the die (105) (base die) adjacent (underneath) to the memory layer (115) includes compute elements configured to perform compute functions on data transmitted from the memory layer (115), particularly from the corresponding memory bank module (116) stacked thereon. Each layer (compute layer (110) and memory layer (115)) can be implemented as a separate die (e.g., each die produced from a single wafer), and thus the CMS device (100) may include a number of adjacent or stacked dies.In the example of FIG. 1a, the die implementing the memory layer (115) can be divided into an array of separate memory bank modules (116) that are directly accessible to compute elements placed adjacent to (e.g., immediately below) them within the compute layer (110), where the lower compute layer (110) is implemented by a die (105) located at the base.

[0046] The CMS device (100) includes a plurality of Through-Silicon Via (152) distributed across multiple regions of the die (105) to provide physical and / or communication connections between a plurality of processing elements (150) and to directly interconnect the compute layer (110) and memory layer (115) of each processing element (150) to support compute functions. The TSVs (152) are vertically positioned between individual elements of the compute layer (110) and memory layer (115) to enable "vertical" communication that can be made directly between a memory bank module (116) separated from each processing element (150) and compute elements positioned below the compute layer (110) (e.g., the memory bank module (116) is stacked on top of the corresponding compute elements). For example, a CMS device (100) may have an architecture comprising a die (105) as a first die implementing a compute layer (110) (e.g., placed in the base, lower layer of the architecture) and at least a second die implementing a memory layer (115) placed on top of the first die (e.g., one or more additional dies stacked on top of the die (105)). The first die may include a plurality of TSVs (152), which may be distributed throughout the die (105) (e.g., separate sets of TSVs (152) placed in different regions / sections of the die (105)). Additionally, the first die may include at least a first compute component and a second compute component (e.g., corresponding to two separate processing elements (150) in an array) implemented on the first die (within the compute layer (110) of the architecture). The second die may include at least a first memory bank module (116) and a second memory bank module (116) (e.g., corresponding to two separate processing elements (150) in the array).The first memory bank module may be connected to the first compute component using a first set of TSVs (152) associated with the first compute component. For example, the first set of TSVs (152) may be associated with the first compute component by being located closer to the first compute component (e.g., contact, connection, etc.). The second memory bank module may be connected to the second compute component using a second set of TSVs (152) associated with the second compute component. For example, the second set of TSVs (152) may be associated with the second compute component by being located closer to the second compute component (e.g., contact, connection, etc.). The second set of TSVs associated with the second compute component may be different from the TSVs (152) corresponding to the first set of TSVs (152). Accordingly, the TSV (152) is configured to transfer data to at least a second die that can be stacked on the lower compute layer (110) of the architecture and the upper memory layer (115) of the architecture (e.g., data stored in a memory bank module (116) is transferred to a corresponding compute component placed below). Here, a set of TSVs (152) functions for each individual processing element (150) (and its components).

[0047] Additionally, when a plurality of processing elements (150) are arranged in an array, a plurality of interconnect ports (154) within the die (105) (base die) are arranged so as to be horizontally positioned between the plurality of processing elements (150), thereby providing physical and / or communication connections between the plurality of processing elements (150), which enables the CMS device (100) to function as a distributed node network with the die (105) as the base die (e.g., each node having memory and computation functions). FIG. 1a illustrates a CMS device (100) composed of a 4x4 array of processing elements (150) implemented in the die (105) (base die), but embodiments of the present invention are not limited thereto, and depending on various embodiments, additional (or fewer) processing elements (150) may be included (e.g., as a larger array and / or an additional layer of processing elements (150). Additionally, according to various embodiments, as disclosed in this specification, additional memory layers (115) (e.g., 8, 12, 16, 24, 32, etc.) may be included in the architecture of the CMS device (100), which may be implemented as additional dies (placed on the base die (105)).

[0048] In one or more embodiments, the configuration of the die (105) (e.g., a base die and upper layer elements implemented thereon) may be considered as a "core" CMS die design for various CMS devices and / or hardware architectures disclosed herein. That is, the die (105) may be used as a "core" stand-alone module that can be repeatedly connected to finally assemble relatively larger and / or more complex CMS devices and / or hardware architectures. For example, examples of CMS devices described in more detail in FIGS. 4 through 6 below may include architectures comprising multiple modularly repeated CMS "core" dies (105) within larger assemblies configured in various ways. Thus, in one or more embodiments, the die (105) may be mounted on a circuit board together with other circuits, external components and / or semiconductor devices to form a CMS device (100). In some embodiments, the CMS device (100) may be implemented as a hardware device or a combination of hardware and / or software components. In some embodiments, the die (105) may be included in a semiconductor microchip and / or semiconductor package that may include additional dies, circuits, external components, semiconductor devices, external pins, pads, electrical connections and / or similar components that may be encased in a protective package. In one or more embodiments, the die (105) may include additional circuitry to support aspects and / or functions of the memory layer (115) and / or compute layer (110). In one or more embodiments, the CMS device (100) may be implemented as a high-bandwidth and / or high-efficiency hardware component of a computer processor, such as a CPU, GPU, NPU (neural processing unit) and / or accelerator processor.For example, an example of a CMS device (100) implementing an accelerator processor is described in more detail with reference to FIG. 8, which can be utilized for data-intensive tasks including AI applications.

[0049] The memory layer (115) may be configured to include circuits for a plurality of memory bank modules (116). As mentioned in this specification, a "memory bank module" may refer to a memory logic unit (e.g., a die) on a physical circuit and / or hardware, which enables high-bandwidth and high-capacity memory functions by allowing parallel access and operation to data as disclosed in this specification. As illustrated in FIG. 1a, the hardware layer (115) may be configured as an array of memory bank modules (116), thereby allowing the memory bank modules (116) to be accessed simultaneously and enabling faster data transfer speeds. The CMS device (100) of FIG. 1a may be configured as an array of 4x4 memory "banks," and a total of 16 memory bank modules (116) are implemented on the die (105). In other words, the CMS device (100) of FIG. 1a may be composed of processing elements (150) in a 4x4 array, and each processing element (150) may be configured to include a corresponding memory bank module (116) implemented thereon. In one or more embodiments, the architecture of the CMS device (100) may be made flexible and / or scalable by implementing a varying number of stacked memory bank modules (116) in the memory layer (115). For example, FIG. 3 shows an example of another CMS device (300) configured separately so that the memory layer (310) includes a 4-stage memory bank stack, wherein the 4-stage memory bank stack may provide improved memory bandwidth and / or capacity compared to the "single memory stack" CMS device (100) implemented in FIG. 1a in such a way that four memory bank modules (316) are vertically stacked per processing element (350). In one or more embodiments, the memory bank modules (316) may be implemented as DRAM dies. Thus, a 4-stage memory bank stack may be implemented as four DRAM dies stacked vertically on top of each other.Although one or more exemplary embodiments of the present disclosure have been described with reference to the CMS device architecture, those skilled in the art will readily understand that many modifications to the exemplary embodiments are possible without substantially departing from the aspects of the present disclosure. Accordingly, all such variations, such as adaptations to the number of vertically stacked DRAM dies that can be implemented within the CMS hardware architecture (e.g., more or fewer memory layers), are intended to be included within the scope of the present disclosure.

[0050] As will be described in more detail in the present disclosure, FIGS. 4 through 6 illustrate examples of different CMS device architectures, and various configurations of the CMS device (e.g., number of different memory bank stacks and / or number of dies) may enable the hardware to adapt and / or optimize to provide desired memory and / or compute functions that are deemed necessary and / or appropriate based, for example, on an AI application.

[0051] Referring again to FIG. 1a, the memory layer (115) may be positioned adjacent to (e.g., on top of) the compute layer (110) at the base of the die (105), and the memory layer (115) may be physically and / or telecommunicationally connected to the compute layer (110) using vias indicated by TSVs (152). As previously described, the architecture of the exemplary CMS device (100) may be configured as a 4x4 array of multiple processing elements (150). Each processing element (150) may have corresponding vertical connections (e.g., interconnections between the memory layer (115) and the compute layer (110)) using TSVs (152), and thus, by utilizing the die (105) as a base die to place multiple connected processing elements (150), multiple TSVs (152) may be distributed across multiple regions of the die (105) (e.g., in contrast to a method of placing TSVs in a centralized region of the die). One aspect of the CMS device (100) and / or hardware architecture disclosed in this disclosure may involve distributing multiple TSVs (152) across the entire die configuration. For example, by implementing multiple processing elements (150) (including corresponding compute and stacked DRAM) on a single die such as the die (105), the architecture may include multiple TSV regions distributed across various regions of the die area (in contrast to a method of placing all TSVs in the center of the die). Accordingly, the disclosed CMS hardware utilizes a distinct arrangement of TSVs (152) distributed across the die (105) (or multi-stacked die) to implement short-range high-bandwidth interconnects between stacked memory bank modules (116) and underlying compute modules (within the compute layer (110)) in a way that can reduce power consumption and mitigate global physical bus and / or global addressability.Additionally, each processing element (150) may have corresponding horizontal connections (e.g., interconnecting multiple processing elements (150) on the die (105)) using interconnection ports (154). Thus, multiple processing elements (150) in a 4x4 array may have physical and / or communication connections between them using distributed TSVs (152), interconnection ports (154), and software (e.g., in contrast to a global physical bus) to support data transmission and / or communication between a distributed network of processing elements (150) on the die (105). Thus, the compute layer (110) may be composed of computing components contained within a processing element (150) "building block" for hardware, which is repeated 16 times on the die (105). This may be referred to as an array of building blocks or an array of processing element building blocks. That is, in the example of FIG. 1a, each of the 16 processing elements (150) of a 4x4 array has a memory bank module (116) (in the memory layer (115)) on top of the die (105), which corresponds to compute components placed directly below the memory bank module (116) (in the compute layer (110)) and stacked thereon to correspond to the memory bank module. Although an array of 16 building blocks is shown in FIG. 1a, the invention is not limited thereto, and the array may include more or fewer building blocks, such as, for example, 6, 8, 9, 12, or 25 building blocks. Details regarding the structure and function of the compute components that may be included in the processing element (150) are described in more detail with reference to FIG. 1b. In one or more embodiments, the compute layer (110) may be implemented as a logic die.Accordingly, in one or more embodiments, the CMS device (100) may be implemented to include a stack of DRAM dies stacked on top of each other and a logic die placed below the stack of DRAM dies.

[0052] The compute layer (110) may be configured to include compute circuits for implementing compute functions, processing and / or computation-related functions that enable the CMS device (100) to function as a high-bandwidth memory and active compute device. The compute layer (110) may be configured to implement various functions identical (or substantially similar) to the functions of the memory utilization component, including but not limited to a host, CPU, GPU, NPU, ASIC, FPGA and / or similar components. For example, the compute circuits of the compute layer (110) may be configured to receive data from the memory layer (115) and perform general or special logic functions on said data, which may be specialized for machine learning and / or AI applications that may have special high-bandwidth requirements. In one or more embodiments, the compute layer (110) may be configured to perform control operations related to basic input / output (I / O) operations and / or communication and / or data transmission with other elements (e.g., memory bank module (116)).

[0053] Additionally, the compute layer (110) may include a plurality of interconnect ports (154) that provide physical and / or communication connections between a plurality of processing elements (150) on the die (105). For example, each processing element (150) is configured to include one or more interconnect ports (154) that can be utilized as a "horizontal" connection with an adjacent processing element (150) in a 4x4 array configuration example. By arranging a plurality of processing elements (150) (each processing element (150) has a corresponding interconnect port (154)) within the array, a mesh network of interconnect ports (154) between the processing elements (150) can be distributed on the compute layer (110), which enables communication between each processing element (150) within the array (e.g., nodes of the mesh network). A network of interconnect ports (154) on the die (105) (distributed on the compute layer (110)) enables multiple processing elements (150) to communicate with each other (and with connected devices) on the die (105), thereby forming an integrated “compute network” system composed of multiple processing elements (150), so as to collectively implement compute functions, processing and / or computation-related capabilities. In one or more embodiments, the compute layer (110) may include multiple ingress / egress inter-die ports (154) that may be configured as horizontal electrical connections between processing elements (150) that may be distributed across separate dies. For example, one or more input / output die ports (154) placed on the die (105) can be connected to other processing elements (150) implemented on separate adjacent dies (e.g., multi-die package of FIG. 5).The input / output die ports (154) provide physical and / or communication interconnections to processing elements (150) on the die (105) and other dies, thereby supporting inter-die communication associated with a large-scale multi-die distributed processing system.

[0054] As can be seen in FIG. 1a, the hardware architecture of the CMS device (100) places compute components (within the compute layer (110)) physically adjacent (e.g., below) and close to the related memory components (within the memory layer (115)) on the die (105), thereby reducing the distance associated with data transfer for performing compute functions. The CMS device (100) utilizes a unique architecture that implements an active compute layer (110) below the memory layer (115) to provide processor-near-memory (PNM) functions that can improve memory bandwidth and performance of specialized hardware, such as machine learning accelerators, while simultaneously lowering energy consumption associated with the die (105). Machine learning and / or AI-based algorithms can benefit from the low latency and improved memory traffic realized by the architecture of the CMS device (100). This is because such applications may require intensive bandwidth and compute efficiency for training and prediction.

[0055] FIG. 1a illustrates that a CMS device (100) may include components that are connected to a die (105) and implement aspects and / or functions related to the high-bandwidth memory and / or compute capabilities of the CMS device (100). In this example, the components may include a controller (121) and an input / output interface (122). The controller (121) may be configured to perform functions related to the control of the system (100) (including functional fallback) and additional compute operations (e.g., high-precision mathematics) that may be implemented by the CMS device (100). In one or more embodiments, the controller (121) may be configured to perform basic operations utilizing RISC (Reduced Instruction Set Computing)-V and to control the operation of the CMS device (100) hardware and / or software, such as the execution of instructions and / or data processing. In one or more embodiments, the controller (121) may be implemented as a processor, microprocessor, CPU and / or similar for the die (105).

[0056] The input / output (I / O) interface (122) may be configured to implement interface-related functions of data and / or components on the die (105), in a manner that enables efficient communication between the die (105) and peripheral devices that can be connected to the die (105). In one or more embodiments, the I / O interface (122) may be configured to implement encryption / decryption, ASIC-related functions (e.g., model IP, protection of user privacy and / or specialized / application-specific end-user programs), specialized I / O-related functions, and others. In one or more embodiments, the I / O interface (122) may be implemented as a PCIe (Peripheral Component Interconnect Express) interface, and the circuitry of the CMS device (100) may be implemented on a PCI-E compatible board.

[0057] FIG. 1b illustrates an exemplary microarchitecture of a processing element (150). As can be seen in FIG. 1b, the processing element (150) may be configured to include one or more components within the microarchitecture, including but not limited to: a memory bank module (116); compute circuits (151); TSVs (152); static memory (153); and interconnect ports (154). As used in this disclosure, the terms “compute” and / or “compute circuits” may refer to computational hardware resources (e.g., circuits, IC modules, logic units, CPUs, GPUs, specialized hardware, etc.) that can be utilized for computations such as training, retraining and / or execution of AI models and / or algorithms.

[0058] The memory bank module (116) may be implemented in the form of dynamic-state volatile computer memory that retains data using power and may have a relatively high capacity and / or speed that is deemed suitable for providing high-bandwidth memory and data storage functions to the CMS device (100). The memory bank module (116) may be implemented as DRAM, SRAM (Static Random Access Memory), MRAM (Magnetoresistive Random Access Memory), or other volatile or non-volatile memory, memory cells, circuits and / or banks, arrays and / or stacked memory configurations suitable and / or other memory components suitable for arraying. In one or more embodiments, the memory bank module (116) is implemented as a DRAM die. Accordingly, in one or more embodiments, the stacking of memory bank modules (116) within the memory layer (115) may be implemented as a plurality of vertically stacked DRAM dies.

[0059] As previously described, the microarchitecture of the processing element (150) may also be described as having a memory layer (115) and a compute layer (110) placed adjacent to (e.g., below) the memory layer (115). The compute layer (110) of the processing element (150) may include several components including compute circuits (151), TSVs (152), static memory (153), and interconnect ports (154), which work together to implement a distributed computing system located directly below the stack of memory bank modules (116) within the memory layer (115).

[0060] The compute circuit (151) may be a circuit configured to implement compute functions and / or performance for the CMS device (100). As mentioned above, the computational operations and / or functions executed by the compute circuit (151) may be performed in close proximity to the memory module, memory circuit and / or memory device of the processing element (150). As can be seen in FIG. 1b, the compute circuit (151) (in the compute layer (110)) may execute compute functions and may be placed directly below and connected to the memory bank module (116) (in the memory layer (115)) to integrate (e.g., embedded) the compute capabilities physically near (e.g., adjacent to the memory bank module (116)) and / or internally (e.g., on a single die) the memory hardware of the CMS device (100). In the example of FIG. 1b, the memory bank module (116) can directly access the compute circuit (151) located at the bottom.

[0061] Accordingly, thanks to proximity, high-bandwidth data transfer between the memory bank module (116) and the compute circuit (151) can be supported in a way that reduces latency over short distances, lowers energy consumption, and minimizes the distance associated with data transfer (e.g., data transfer between separate processing units of the memory and processing element (150). For example, the compute circuit (151) may be configured to receive data from the memory bank module (116) and perform one or more compute functions, which may be mathematical operations that may be included in AI and / or machine learning tasks. In one or more embodiments, the compute circuit (151) may perform compute functions, instructions, and / or tasks associated with machine learning operations wholly and / or partially, which include, but are not limited to, matrix multiplication, dot products, and activation functions used in neural networks. In one or more embodiments, the compute circuit (151) may be configured to include logic to support various controller functions for the CMS device (100), such as coordinating data transfer between the memory bank module (116) and other components of the processing element (150), scheduling tasks, and managing data flow, which may be done in a manner that enables an increase in throughput and / or a reduction in latency of the CMS device (100). Thus, by using TSVs (152) to directly connect the compute circuit (151) to the memory bank module (116) and utilizing software (in contrast to the global physical bus of HBM) for the transfer of data between them, the processing element (150) may provide memory and integrated active compute components, thereby significantly improving the performance and / or energy efficiency of the CMS device (100) for data-intensive tasks, such as AI applications, for example.

[0062] TSVs (152) may be configured as high-performance interconnections between the memory layer (115) (and components within it) and the compute layer (110) (and components within it). The TSVs (152) may pass through the die (105) to enable vertical electrical connections (vias) that form high-bandwidth interconnections for 3D (3-dimensional) stacked CMS dies, devices, and / or packages as disclosed in this disclosure. In one or more embodiments, the TSVs (152) may be vertical wires connecting different layers of the microchip and / or stacked dies, thereby enabling direct die-to-die communication. In one or more embodiments, a plurality of TSVs (152) may be implemented as data TSVs configured to transmit data signals between different layers and / or dies and / or power TSVs configured to transmit power signals and ground connections, which ensures efficient power supply to different parts of the die or stacked dies. For example, TSVs (152) can provide communication connections between multi-stacked DRAM dies in the memory layer (115) and compute dies in the compute layer (110). Accordingly, TSVs (152) can enable short-distance and high-bandwidth data transmission between the memory bank module (116) and the compute circuit (151) for the CMS device (100) in a manner that reduces latency and / or lowers power consumption (e.g., elimination of the global data bus, elimination of long-distance and / or high-power consumption data transfer from memory stacks to the SoC for computing in existing HBM, etc.).

[0063] The processing element (150) may be configured to include static memory (153). Static memory (153) may be implemented as SRAM that is determined to be suitable for providing cache and / or high-speed register capabilities for the CMS device (100), for example, by retaining data while power is supplied and having relatively fast access times and low latency (e.g., relative to the memory bank module (116)). Static memory (153) may be implemented as SRAM, memory cells, circuits and / or other memory components that are determined to be suitable and / or appropriate for working with the compute circuit (151). In one or more embodiments, static memory (153) is implemented as single-ported (e.g., one data port for reading and writing at a time) SRAM.

[0064] A processing element (150) may be configured to include interconnect port(s) (154). In one or more embodiments, the interconnect port(s) (154) may be implemented as horizontal electrical connections for input / output connections to one or more other processing elements (150). For example, the processing element (150) may use the interconnect ports (154) to serve as physical connections to one or more processing elements placed adjacent to the processing element (150) on the die (105) in the 4x4 array configuration of FIG. 1a. Additionally, placing multiple processing elements (150) together on the die (105) may include multiple interconnect ports (154) distributed over the die (105), and may form, for example, a "mesh" type network for communication between the processing elements (150). For example, each processing element (150) arranged in a 4x4 array on the die (105) may include corresponding interconnect ports (154) that horizontally connect each processing element (150) to one or more adjacent processing elements (150) on the die (015). The interconnect ports (154) are configured to support inter-processor connections and / or communication between multiple processing elements (105). Thus, multiple interconnected processing elements (150) can collectively operate as a distributed network of computing nodes through the interconnect ports (154) connected on the die (105). In one or more embodiments, the microarchitecture of the processing element (150) may have variations in the number and / or configuration of the interconnect port(s) (154) as determined to be suitable for the specific architecture of the CMS core die, device and / or package.

[0065] As previously mentioned, AI workloads may include vast data sets and may require thousands of operations. Therefore, the processing element (150) can be utilized as a core unit that provides both memory and compute processing capabilities in a manner suitable or optimized for scalability and parallelism (e.g., simultaneous execution of multiple tasks), ultimately enabling the CMS device (100) (consisting of an array of processing elements (150)) to become efficient hardware for AI applications.

[0066] FIG. 2a is a two-dimensional (2D) top view of an exemplary CMS die (200). The structure, components, and functions of the CMS die (200) are substantially similar to the CMS "core" die (105) described earlier with reference to FIG. 1a and FIG. 1b. Therefore, for brevity, some details of the CMS die (200) may not be repeated in FIG. 2a. FIG. 2a shows that the CMS die (200) may have an architecture comprising a plurality of stacked memory bank modules (216) with a plurality of TSVs (252) distributed among them. The TSVs (252) may be distributed within the CMS die (200) to implement short-range high-bandwidth interconnects between the stacked memory bank modules (216) and lower computing modules in a manner that can reduce power consumption and mitigate global physical bus and / or global addressability. In one or more embodiments, the CMS device may be added to a physical bus structure (e.g., see FIG. 2b) or, instead, utilize software to manage data movement between processing elements (e.g., compute, memory, etc.) within a distributed computing system (e.g., message-passing). For example, communication between multiple processing elements on a die may be controlled and / or directed by software and may be implemented through physical connections between processing elements established by interconnection ports.

[0067] FIG. 2b illustrates another exemplary architecture of a CMS device (260) according to some embodiments. In one or more embodiments, the components of the CMS device (260) may be implemented on a single die. The CMS device (260) may be configured to include a first plurality of stacked memory bank modules (261a) having corresponding sub-logic modules (262a) disposed in opposing regions (260a, 260b) of the die, and a second plurality of stacked memory bank modules (261b) having corresponding sub-logic modules (262b). In one or more embodiments, the plurality of stacked memory bank modules (261a, 261b) may be implemented with multiple stacked DRAM dies, and the logic modules (262a, 262b) may be implemented with logic dies disposed below the corresponding memory bank modules (261a, 261b).

[0068] As can be seen in FIG. 2b, a first plurality of stacked memory bank modules (261a) and logic modules (262a) are positioned within a region (260a) located on one side of the die (e.g., near the first periphery and / or edge of the die), a second plurality of stacked memory bank modules (261b) and logic modules (262b) are positioned within a region (260b) located on the opposite side of the die (e.g., near the second periphery of the die), and a plurality of TSVs (263) are positioned at a centralized location of the die and located between them. For example, the CMS device (260) may include an "on-die" host that can be configured to perform compute functions as disclosed in the present disclosure. The host may be positioned substantially close to the center of the die, for example, physically adjacent to an area of ​​the die containing a plurality of TSVs (263) between both sides of the stacked memory bank modules (261a, 261b). Accordingly, the CMS device (260) may include a plurality of physical buses (264) for transferring data (e.g. for compute processing) from a plurality of stacked memory bank modules (261a, 261b) and logic modules (262a, 262b) to the host using the centrally located TSVs (263). Accordingly, data that may be stored in one of the first plurality of stacked memory bank modules (261a) and / or logic modules (262a) may have to travel a distance until it is received by the TSVs (263) in the area (260a) to be transferred to a separate host for compute processing. As mentioned above, the CMS device (200) (e.g., see FIG. 2a) is composed of TSVs distributed across the entire die and is configured to mitigate the use of the global physical bus by utilizing software for data communication (e.g., message delivery).

[0069] FIG. 3 illustrates an example of another CMS device (300) that may be configured to implement high-bandwidth memory and compute processing capabilities as disclosed in the present disclosure. In general description, the CMS device (300) may be described as a high-capacity variant of the CMS device (100) implemented on a single die (305) (e.g., see FIG. 1a). The architecture, components, and functions of the CMS device (300) are substantially similar to those of the CMS device (100) described earlier with reference to FIG. 1a and FIG. 1b. Therefore, for brevity, FIG. 3 may not repeat some details of the CMS device (300). However, FIG. 3 shows that the architecture of the CMS device (300) may include a four-tier (i.e., four-layer) stacked configuration comprising a plurality of vertically stacked memory bank modules (316) instead of the "single stack" memory bank modules (116) used in the CMS device (100) (e.g., see FIG. 1a). FIG. 3 illustrates the architecture of the CMS device (300) having four layers of vertically stacked memory bank modules, but embodiments according to the present disclosure are not limited thereto, and according to some embodiments, the CMS device (300) may include a plurality of layers of vertically stacked memory bank modules that are less than, equal to or greater than the number shown in FIG. 3.

[0070] As illustrated in FIG. 3, the CMS device (300) may be composed of a 4x4 array of processing elements (350). Each processing element (350) may include four memory bank modules (316) stacked on top of each other within a memory layer (315), and a compute layer (310) (and its components) is placed below the four-layer stack of memory bank modules (316). That is, an exemplary CMS device (300) may include an NxM (e.g., 4x4) array of processing elements (350), and each processing element (350) includes four-layer stacked memory bank modules (316) and a corresponding compute layer (310) underneath. Accordingly, the CMS device (300) may have an architecture including increased memory hardware compared to the CMS device (100) described in FIG. 1a, thereby allowing the memory bandwidth and capacity of the CMS device (300) to be expanded, for example, for enhanced performance and / or more complex AI applications. In one or more embodiments, the four-layer stacking of memory bank modules (316) may be implemented with four stacked DRAM dies. Accordingly, in one or more embodiments, the CMS device (300) may include a four-layer stacking of a plurality of stacked DRAM dies additionally stacked on top of a logic die.

[0071] FIG. 4 illustrates another example of a CMS device (400) that may be configured to implement high-bandwidth memory and compute processing capabilities as disclosed herein. In general, the CMS device (400) may be described as an increased capacity variant of the CMS device (100) (e.g., see FIG. 1a) and the CMS device (300) (e.g., see FIG. 3). The architecture, components, and functions of the CMS device (400) are substantially similar to the CMS device (100) described earlier with reference to FIG. 1a and FIG. 1b. Therefore, for brevity, details of the CMS device (400) are not described again with reference to FIG. 4. However, FIG. 4 illustrates that the architecture of the CMS device (400) may include an 8-tier stack configuration comprising multiple memory bank modules (416) stacked vertically eight times, instead of a "single stack" of memory bank modules (116) used in the CMS device (100) (e.g., see FIG. 1a). In a 4x4 array of processing elements (450) of the CMS device (400), each processing element (450) may include eight memory bank modules (416) stacked on top of each other, and the corresponding compute layer (410) (and its components) is positioned below the 8-tier stack of memory bank modules (316). That is, the CMS device (400) may include a 4x4 array of processing elements (450), and each processing element (450) includes an 8-layer stacked memory bank module (416) with a corresponding compute layer (410) placed at the bottom. In one or more embodiments, the 8-layer stacking of the memory bank module (416) may be implemented with 8 stacked DRAM dies. Accordingly, in one or more embodiments, the CMS device (400) may include an 8-layer stacking of a plurality of stacked DRAM dies additionally stacked on top of a logic die.

[0072] FIG. 5 illustrates another example of a CMS device (500) that may be configured to implement high-bandwidth memory and compute processing capabilities as disclosed herein. The architecture of the CMS device (500) may include a modular assembly of CMS devices (e.g., 100, 300, 400). For example, the CMS device (500) may be implemented as a multi-die semiconductor package, which may be configured to provide high-level distributed computing by including a 4x4 array of multiple modularly connected CMS devices (300) (e.g., see FIG. 3). Each processing element on the corresponding CMS device (300) may include input / output die-to-die ports configured to support horizontal electrical connections to other processing elements that may be distributed on separate dies. Accordingly, by arranging multiple CMS devices (300) each having multiple input / output die-to-die ports, die-to-die communication is supported across a multi-die semiconductor package including a 4x4 array of multiple modularly connected CMS devices (300).

[0073] The architecture of the CMS device (500) may be configured such that each layer is composed of a single die, as shown in FIG. 5, for example, depending on the scale / size, architecture, and / or desired application of the CMS device (500). However, in one or more embodiments, the CMS device (500) may be a package comprising individual layers composed of hardware larger than a single die. For example, it may include individual units (e.g., 100) packaged together on an interposer and / or PCB, and may have additional management / communication components within the package. Thus, the CMS device (500) may include multiple sub-packages connected within a single package, and may include, for example, an interposer that acts as a bridge for signals and power between the sub-packages and the package substrate. Each of the sub-packages within the CMS device (500) may be implemented as a modular functional circuit block (e.g., independently designed and manufactured) and then assembled into an interposer, or packaged into sub-units and then further packaged into an interposer, which may provide several advantages over conventional monolithic SoCs (System-on-Chip).

[0074] The structure, components, and functions of the individual components of the CMS device (500) are identical (or substantially similar) to the CMS device (100) previously described with reference to FIG. 1a and / or the CMS device (300) previously described with reference to FIG. 3. Therefore, for brevity, some details of the CMS device (500) may not be described again with reference to FIG. 5. However, FIG. 5 shows that the architecture of the CMS device (500) may be a larger assembly that expands the CMS device 500 into variations of higher performance and larger capacity by modularly repeating and connecting a number of small CMS "core" dies (e.g., CMS device (300)) (e.g., via inter-die input / output ports). In some embodiments, the CMS device (500) may be implemented as a semiconductor microchip and / or semiconductor package that may include additional dies, circuits, external components, semiconductor devices, external pins, pads, electrical connections and / or similar things that may be embedded in a protection package.

[0075] Accordingly, the CMS dies, devices, and packages disclosed herein can be utilized to provide high-bandwidth memory and compute processing capabilities in a modular and scalable manner so as to be optimized based on the complexity and processing requirements of the application. For example, the CMS device (100) (e.g., see FIG. 1) can be utilized as a low-cost variant of an on-device AI-based processor, and the CMS device (500) can be utilized as a high-performance, high-capacity "super chip" variant for large-scale AI-based applications such as image and voice classification.

[0076] FIG. 6 is a flowchart illustrating exemplary operations of a method for transmitting data within the architecture of the CMS device (100) of FIG. 1a according to some embodiments of the present disclosure. For example, FIG. 6 illustrates various operations of a method (600) that may be implemented by software for supporting, managing, and / or controlling communication (including data transmission) between various components within the architecture of the CMS device (100) that may be enabled through a plurality of TSVs (152) (e.g., see FIG. 1a). In one or more embodiments, the software may include logic and / or instructions implemented by one or more components of the CMS device (100), such as a controller (121), a memory bank module (116), a compute circuit (151), static memory (153), etc. In one or more embodiments, the software may include logic and / or instructions implemented outside the CMS device (100), such as a host, a CPU, a controller, etc. FIG. 6 illustrates various operations of a method according to some embodiments, but embodiments of the present disclosure are not limited thereto and may include additional operations or fewer operations according to various embodiments without departing from the spirit and scope of the embodiments of the present invention.

[0077] Additionally, there may be software utilized to support communication between various components of the CMS device (100) that may be involved in the execution of in-memory compute functions (replacing the functions of a physical bus). In one or more embodiments, the CMS device (100) may be configured to utilize software to perform and / or execute various operations, including but not limited to the following: data movement and / or transmission control, message delivery, data preparation and layout execution, task scheduling and synchronization execution, memory controller function execution, operation initiation, memory layout management, and / or similar operations.

[0078] Referring to FIG. 6, the method (600) may include one or more of the following operations. A memory bank module may be allocated (operation 605). In an example of operation, the CMS device (100) may be used by a computer device to perform computational functions related to an AI application (e.g., multiplication of two vectors). Data that may be involved in computational functions may not initially be located within the memory layer (115) of the CMS device (100). For example, data representing vector A and vector B to be multiplied may be stored in components and / or devices outside the CMS device (100), such as a CPU, main memory, GPU, etc. Accordingly, messages and / or instructions may be delivered to multiple memory bank modules (116) within the memory layer (115) of the CMS device (100) to allocate at least one memory bank module (116) to be utilized for computational functions. For example, a memory bank module (116) may be allocated to store data during the execution of computational functions. In one or more embodiments, messages and / or instructions may be dispatched via TSVs to a plurality of memory bank modules (116) within an NxN array of processing elements (150) to allocate a selected memory bank module (116) (e.g., corresponding to a processing element (150)). In one or more embodiments, the allocation of a memory bank module (116) may include reserving, allocating, and / or accessing a specific portion of memory within the memory bank module (116) for use in a specific purpose or task to enable systematic and efficient use of available memory resources. The allocation of the memory bank module (116) may be dynamic, and in some embodiments, the memory bank modules (116) are allocated and / or reconfigured based on changing workload requirements. Or in one or more embodiments, the allocation may be static, and the memory bank modules (116) are predefined and / or allocated for a specific task.

[0079] Data may be transmitted to an allocated memory bank module (116) (operation 610). Messages, instructions, and / or data may be transmitted to a memory bank module (116) within the memory layer (115) of the CMS device (100) to control data movement to allocated memory bank modules (116) to be utilized for compute functions. For example, data representing vector A and vector B may be transmitted to an allocated memory bank module (116) to store data during vector multiplication execution (e.g., from a CPU outside the CMS device (100)). In one or more embodiments, messages and / or instructions may be dispatched via TSVs (152) to a memory bank module (116) located in a processing element (150) of an NxN array to transmit and / or store data in the allocated memory bank module (116) (e.g., corresponding to the processing element (150)). Therefore, efficient data transfer for the allocation and / or operation functions of the memory bank module (116) (e.g., vector A and vector B for vector multiplication) can be synchronized and / or controlled via software (e.g., without a bus).

[0080] A compute function may be triggered (operation 615). Messages and / or instructions may be transmitted to a compute circuit (151) corresponding to a memory bank module (116) (e.g., for a processing element (150)) to initiate and / or control the circuit to execute a computation function. For example, the compute circuit (151) may be configured to execute an operation related to performing vector multiplication of vector A and vector B, wherein data representing the vectors is stored in the memory bank module (116) during operations. In one or more embodiments, messages, instructions and / or data may be communicated between the compute circuit (151) and the corresponding memory bank module (116) via TSVs (152). A memory bank module (116) and / or compute circuit (151) may be configured to transmit processing instructions to a corresponding compute circuit (151) placed below a specific memory bank module (116) that stores data related to these instructions (e.g., on the same processing element). In one or more embodiments, an on-die controller (or other components) may implement data transfer management between processing elements and / or layers (on each processing element) placed on the die. Accordingly, the controller and / or related software may be configured to control the communication of processing instructions to a compute circuit (151) appropriately corresponding to the memory bank module (116) in use in the operation.

[0081] The compute circuit (151) is capable of executing compute functions and is positioned directly below the memory bank module (116). Additionally, the compute circuit (151), which is connected to the memory bank module (116) using TSVs (152), provides physical and communication interconnections, which can improve performance and energy efficiency (e.g., elimination of physical buses, reduction of data movement, etc.) in a manner that allows compute capabilities to be integrated physically near (e.g., adjacent to the memory bank module (116)) and / or internally (e.g., on a single die) the memory hardware of the CMS device (100). Thus, the execution of compute functions (e.g., vector multiplication) can be triggered, performed, and / or controlled by the compute circuit (151) via software (e.g., without a bus).

[0082] The results of the computational function may be transmitted to the memory bank module (operation 620). Messages, data, and / or instructions may be transmitted to the memory bank module (116) after the compute circuit (151) has executed operations related to the computational function. For example, the compute circuit (151) may complete operations to generate the result of vector multiplication of vector A and vector B. In one or more embodiments, messages, instructions, and / or data may be communicated between the compute circuit (151) and the corresponding memory bank module (116) via TSVs (152) to transmit and / or store the results of the computational function in the memory bank module. Thus, the execution of the computational function (e.g., vector multiplication) may be performed by the memory bank module (116) via software (e.g., without a bus) and the result may be obtained, which can be done in a manner that reduces data movement and power consumption, and increases the efficiency of tasks that may involve large amounts of memory access, such as AI applications.

[0083] FIG. 7 is a block diagram of an electronic device implementing a parallel accelerator for AI applications utilizing an electronic device, for example, a CMS device (e.g., see FIG. 1), according to some embodiment of the present disclosure. For example, the electronic device (701) may be configured to implement a generative AI application, such as an AI application that generates new content (e.g., text, images, music, videos, etc.) based on patterns learned from vast data sets. Accordingly, the processor (702) may include an accelerator processor and may be implemented as an auxiliary processor (723) configured to be optimized for AI-related computations and other processing tasks, for example. The CMS device may be physically integrated (e.g., included in a processor package) into the auxiliary processor (723) in a manner that can reduce (or minimize) latency and / or improve (or maximize) bandwidth. In some embodiments, the CMS device may be attached to and / or integrated with other components of the electronic device (701) (including, but not limited to, the main processor (721), memory (730), etc.) in addition to and / or replacing the auxiliary processor (723).

[0084] As an operational example, the electronic device (701) can perform complex training processes for large-scale models related to generative AI applications. For example, the electronic device (701) can utilize an auxiliary processor (723) as an accelerator processor to process large matrices of input data, weights, and / or activations related to model training. The auxiliary processor (723), including a CMS device, can improve the overall performance and / or efficiency of the electronic device (701) when implementing AI applications by providing data transmission related to performing data during training (e.g., parameters, training data, etc.) and / or computational tasks related to model training (e.g., matrix multiplication, reading / writing of calculation results, etc.) in a delay-reducing manner.

[0085] Referring to FIG. 7, an electronic device (701) in a network environment (700) may communicate with an electronic device (702) through a first network (e.g., a short-range wireless communication network) or with an electronic device (704) or a server (708) through a second network (e.g., a long-range wireless communication network). The electronic device (701) may communicate with an electronic device (704) through a server (708). The electronic device (701) may include a processor (720), memory (730), input device (750), sound output device (755), display device (760), audio module (770), sensor module (776), interface (777), haptic module (779), camera module (780), power management module (788), battery (789), communication module (790), Subscriber Identity Module (SIM) card (796), or antenna module (797). In one embodiment, at least one of the components (e.g., a display device (760) or a camera module (780)) may be omitted from the electronic device (701), or one or more other components may be added to the electronic device (701). Some components may be implemented as a single integrated circuit (IC). For example, a sensor module (776) (e.g., a fingerprint sensor, an iris sensor, or an ambient light sensor) may be embedded in the display device (760) (e.g., a display).

[0086] The processor (720) can execute software (e.g., program (740)) for controlling at least one other component (e.g., hardware or software component) of the electronic device (701) combined with the processor (720), and can perform various data processing or operations.

[0087] As at least part of the data processing or computation, the processor (720) may load commands or data received from other components (e.g., sensor module (776) or communication module (790) into volatile memory (732), process the commands or data stored in volatile memory (732), and store the resulting data in non-volatile memory (734). The processor (720) may include a main processor (721) (e.g., central processing unit (CPU) or application processor (AP)) and an auxiliary processor (723) (e.g., graphics processing unit (GPU), image signal processor (ISP), sensor hub processor, or communication processor (CP)) that can operate independently of or together with the main processor (721). Additionally or alternatively, the auxiliary processor (723) may be configured to consume less power than the main processor (721) or to execute specific functions. The auxiliary processor (723) may be separated from the main processor (721) or implemented as part of the main processor.

[0088] The auxiliary processor (723) may control a part of the function or state associated with at least one component of the electronic device (701) (e.g., display device (760), sensor module (776), or communication module (790) on behalf of the main processor (721) while the main processor (721) is in an inactive (e.g., sleep) state, or may operate together with the main processor while the main processor (721) is active (e.g., application execution). The auxiliary processor (e.g., image signal processor or communication processor) may be implemented as part of another component (e.g., camera module (780) or communication module (790)) functionally related to the auxiliary processor (723).

[0089] The memory (730) can store various data used by at least one component of the electronic device (701) (e.g., a processor (720) or a sensor module (776)). The various data may include, for example, input data or output data for software (e.g., a program (740) and related commands. The memory (730) may include volatile memory (732) or non-volatile memory (734). The non-volatile memory (734) may include internal memory (736) and / or external memory (738).

[0090] The program (740) may be stored in memory (730) as software and may include, for example, an operating system (OS) (742), middleware (744), or an application (746).

[0091] The input device (750) can receive commands or data to be used by another component of the electronic device (701) (e.g., processor (720)) from outside the electronic device (701) (e.g., user). The input device (750) may include, for example, a microphone, a mouse, or a keyboard.

[0092] The sound output device (755) can output a sound signal to the outside of the electronic device (701). The sound output device (755) may include, for example, a speaker or a receiver. The speaker may be used for general purposes such as multimedia playback or recording, and the receiver may be used for receiving incoming calls. The receiver may be separate from the speaker or implemented as part of the speaker.

[0093] The display device (760) can provide visual information to an external (e.g., user) of the electronic device (701). The display device (760) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the corresponding display, holographic device, or projector. The display device (1660) may include a touch circuit adjusted to detect a touch or a sensor circuit adjusted to measure the intensity of a force generated by a touch (e.g., a pressure sensor).

[0094] The audio module (770) can convert sound into an electrical signal and vice versa. The audio module (770) can acquire sound through the input device (750) or output sound directly (e.g., wired) or wirelessly combined with the electronic device (701) through the sound output device (755) or headphones of the external electronic device (702).

[0095] The sensor module (776) can detect the operating state of the electronic device (701) (e.g., power or temperature) or the environmental state outside the electronic device (701) (e.g., user state) and then generate an electrical signal or data value corresponding to the detected state. The sensor module (776) may include, for example, a gesture sensor, a gyroscope sensor, an atmospheric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0096] The interface (777) may support one or more specified protocols used to combine the electronic device (701) with an external electronic device (702) directly (e.g., wired) or wirelessly. The interface (777) may include, for example, a High-Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital (SD) card interface, or an audio interface.

[0097] The connection terminal (778) may include a connector that allows the electronic device (701) to be physically connected to an external electronic device (702). The connection terminal (778) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0098] The haptic module (779) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic sensation. The haptic module (779) may include, for example, a motor, a piezoelectric element, or an electric stimulator.

[0099] The camera module (780) can capture a still image or a moving image. The camera module (780) may include one or more lenses, an image sensor, an image signal processor, or a flash. The power management module (788) can manage the power supplied to the electronic device (701). The power management module (788) may be implemented, for example, as at least part of a Power Management Integrated Circuit (PMIC).

[0100] The battery (789) can supply power to at least one component of the electronic device (701). The battery (789) may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

[0101] The communication module (790) can support establishing a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (701) and an external electronic device (e.g., electronic device (702), electronic device (704), or server (708)) and performing communication through the established communication channel. The communication module (790) may include one or more communication processors that can operate independently of the processor (720) (e.g., AP) and support direct (e.g., wired) communication or wireless communication. The communication module (790) may include a wireless communication module (792) (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or a wired communication module (794) (e.g., Local Area Network (LAN) communication module or Power Line Communication (PLC) module). One of these communication modules may communicate with an external electronic device through a first network (798) (e.g., a short-range communication network such as Bluetooth, Wi-Fi Direct, or the Infrared Data Association (IrDA) standard) or a second network (799) (e.g., a cellular network, the Internet, or a computer network such as a long-range communication network such as a LAN or a Wide Area Network (WAN)). These various types of communication modules may be implemented as a single component (e.g., a single IC) or as multiple separate components (e.g., multiple ICs).The wireless communication module (792) can identify and authenticate the electronic device (701) in a communication network such as a first network (798) or a second network (799) using subscriber information (e.g., International Mobile Subscriber Identity, IMSI) stored in the subscriber identification module (796).

[0102] The antenna module (797) can transmit or receive a signal or power to or from outside the electronic device (701) (e.g., an external electronic device). The antenna module (797) may include one or more antennas, and accordingly, at least one antenna suitable for a communication method used in a communication network by the communication module (790) (e.g., a wireless communication module (792)), e.g., a first network (798) or a second network (799), may be selected. Then, the signal or power may be transmitted or received between the communication module (790) and the external electronic device through the selected at least one antenna.

[0103] Commands or data may be transmitted or received between the electronic device (701) and the external electronic device (704) via a server (708) combined with the second network (799). Each electronic device (702 and 704) may be of the same type or a different type as the electronic device (701). All or part of the operation performed on the electronic device (701) may be performed on one or more of the external electronic devices (702, 704, or 708). For example, if the electronic device (701) needs to perform a function or service automatically, or in response to a request from a user or another device, the electronic device (701) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service, or additionally. One or more external electronic devices that receive the request may perform at least part of the requested function or service or additional functions or additional services related to the request and transmit the result of the performance to the electronic device (701). The electronic device (701) may be provided as at least part of the response to the request, regardless of whether the result is further processed. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

[0104] Embodiments of the subject matter and operation described herein may be implemented in digital electronic circuits comprising structures and structural equivalents disclosed herein, or in computer software, firmware, or hardware, or in a combination of one or more of these. Embodiments of the subject matter described herein may be implemented in one or more computer programs, namely, one or more modules of computer program instructions encoded in a computer storage medium for the execution of a data processing device or for controlling the operation of a data processing device. Additionally or alternatively, program instructions may be encoded in artificially generated radio signals, for example, mechanically generated electrical, optical, or electromagnetic signals generated to encode information to be transmitted to a suitable receiving device for execution by the data processing device. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof, or may be included therein. Additionally, although the computer storage medium is not a radio signal, the computer storage medium may be a source or target of computer program instructions encoded in an artificially generated radio signal. A computer storage medium may be one or more individual physical components or media (e.g., multiple CDs, disks, or other storage devices) or may be contained therein. Additionally, the operations described herein may be implemented as operations performed by a data processing device on data stored in one or more computer-readable storage devices or received from other sources.

[0105] While this specification may contain many specific implementation details, such implementation details should not be interpreted as a limitation on the scope of the claims, but rather as a description of features specific to specific embodiments. Specific features described in the context of individual embodiments in this specification may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, even if features are described above as operating in a specific combination and were initially claimed as such, one or more features from the claimed combination may be excluded from the combination as applicable, and the claimed combination may be referred to as a sub-combination or a variation of a sub-combination.

[0106] Likewise, although operations are depicted in a specific order in the drawings, it should not be understood that such operations must be performed in the specific order indicated or sequential order, or that all illustrated operations must be performed, in order to obtain a desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the aforementioned embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0107] Accordingly, specific embodiments of the present invention have been described herein. Other embodiments are within the scope of the following claims. In some cases, the operations specified in the claims may be performed in a different order to achieve desirable results. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific order or sequential order indicated to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0108] As will be recognized by those skilled in the art, the innovative concepts described herein may be modified and varied in a wide range of applications. Accordingly, the scope of the claimed technology is not limited to the specific exemplary teachings discussed above, but is rather defined by the following claims.

Claims

Claim 1 A device comprising: a first die including a first compute component, a second compute component, a first Through Silicon Via (TSV) set connected to the first compute component and connecting the stacked dies of the device, and a second TSV set connected to the second compute component and connecting the stacked dies of the device; and a second die stacked on the first die, wherein the second die comprises a first memory bank module connected to the first compute component using the first TSV set and a second memory bank module connected to the second compute component using the second TSV set. Claim 2 An apparatus according to claim 1, further comprising a third die stacked on the second die, wherein the third die comprises: a third memory bank module connected to the first compute component using the first TSV set; and a fourth memory bank module connected to the second compute component using the second TSV set. Claim 3 A device according to claim 2, wherein the first memory bank module connected to the first compute component using the first TSV set forms a first processing element configured to perform compute functions using data from the first memory bank module, and the second memory bank module connected to the second compute component using the second TSV set forms a second processing element configured to perform compute functions using data from the second memory bank module. Claim 4 An apparatus according to claim 3, further comprising a plurality of interconnected ports of the first die, wherein the ports connect the first processing element and the second processing element to one or more additional processing elements, wherein the first processing element and the second processing element comprise a compute circuit configured to perform a compute function, and the one or more additional processing elements comprise a compute component connected to at least one memory bank module, and wherein the at least one memory bank module is selected from an array of memory bank modules arranged on the first die and stacked on the first die using a corresponding set of TSVs. Claim 5 A device according to claim 4, wherein, for the one or more additional processing elements, each compute component is configured to perform a computational function on data from each of at least one memory bank module stacked thereon. Claim 6 The apparatus of claim 5 further comprises a controller programmed to execute the transmission of data between the first compute component and the first memory bank module, the transmission of data between the second compute component and the second memory bank module, and the transmission of data between each of at least one memory bank module and each of the compute components for each of the one or more additional processing elements. Claim 7 In claim 5, the device wherein the first TSV set and the second TSV set are part of a plurality of TSVs distributed in a plurality of regions of the first die. Claim 8 An apparatus according to claim 7, wherein the first TSV set comprises at least one of the plurality of TSVs located in a first region of the first die in contact with the first compute component, and the second TSV set comprises at least one of the plurality of TSVs located in a second region of the first die in contact with the second compute component. Claim 9 A device according to claim 3, wherein the first processing component further includes the third memory bank module and is configured to perform a compute function using data from the third memory bank module, and the second processing component further includes the fourth memory bank module and is configured to perform a compute function using data from the fourth memory bank module. Claim 10 In paragraph 2, the device comprises a plurality of additional dies stacked on the third die, wherein the stacked dies of the device include a plurality of additional dies stacked on the third die. Claim 11 In paragraph 2, the apparatus further comprises dynamic random access memory (DRAM) dies, wherein one or more of the second die or the third die comprises a DRAM die. Claim 12 In paragraph 3, the device wherein the compute function comprises one or more of matrix multiplication, inner product, activation function, or mathematical function related to an AI (Artificial Intelligence) application. Claim 13 An apparatus comprising: a first die; and a second die stacked on the first die, wherein the first die comprises a plurality of TSVs distributed in a plurality of regions of the first die, a first compute component, a second compute component, a first set of TSVs associated with the first compute component and connecting the stacked dies of the apparatus, and a second set of TSVs associated with the second compute component and connecting the stacked dies of the apparatus, wherein the first set of TSVs comprises one or more of the plurality of TSVs within a first region of the first die, and the second set of TSVs comprises one or more of the plurality of TSVs within a second region different from the first region of the first die, and wherein the second die comprises a first memory bank module connected to the first compute component using the first set of TSVs and a second memory bank module connected to the second compute component using the second set of TSVs. Claim 14 An apparatus according to claim 13, wherein the first memory bank module connected to the first compute component using the first TSV set forms a first processing element configured to perform compute functions using data from the first memory bank module, and the second memory bank module connected to the second compute component using the second TSV set forms a second processing element configured to perform compute functions using data from the second memory bank module. Claim 15 In paragraph 13, the device comprises, wherein the stacked dies of the device include a plurality of additional dies stacked on the second die. Claim 16 In paragraph 15, the apparatus further comprises a dynamic random access memory (DRAM) die, wherein the second die or one or more of the plurality of dies comprises a DRAM die. Claim 17 In claim 16, the apparatus is configured such that the plurality of TSVs transmit data stored in the DRAM dies to the first die. Claim 18 In claim 14, the apparatus further comprises a plurality of interconnection ports on the first die configured to connect the first processing element and the second processing element to a plurality of processing elements in order to implement a distributed computing system. Claim 19 A device according to claim 13, further comprising a controller programmed to execute data transfer between the first compute component and the first memory bank module and data transfer between the second compute component and the second memory bank module. Claim 20 A system comprising: an accelerator processor configured to perform a compute function; and a memory storing instructions that, when executed by the accelerator processor, cause the compute function to be performed, wherein the accelerator processor comprises: a first compute component; a second compute component; a first Through Silicon Via (TSV) set associated with the first compute component and connecting stacked dies of the device; a first die comprising a second TSV set associated with the second compute component and connecting stacked dies of the device; and a second die stacked on the first die, wherein the second die comprises a first memory bank module connected to the first compute component using the first TSV set and a second memory bank module connected to the second compute component using the second TSV set.

Images