Multi-channel memory for augmenting local memory

Abstract

A memory system, a method of assembling the memory system, and a computer system. The memory system includes a global memory device coupled to a plurality of processing elements. The global memory device is placed outside a chip on which the plurality of processing devices reside. The memory system also includes at least one primary scratch pad memory coupled to the at least one processing element of the plurality of processing devices and to the global memory device. The memory system also includes a plurality of auxiliary scratch pad memories coupled to the plurality of processing elements and the global memory device. One or more auxiliary memo memories are configured to store a static tensor. At least a portion of the plurality of auxiliary scratch pad memories is configured as a single multi-channel device.

Description

Background Art

[0001] The present disclosure relates to random access memory (RAM) devices, and more particularly, to increasing the capacity of a primary on-chip scratchpad for a processing core.

[0002] At least some known special-purpose accelerators, such as but not limited to deep learning accelerators, include a set of interconnected processing cores within a multi-core processing device, where each core uses local memory or scratch pad memory to quickly access its working set, and can access global memory to periodically or continuously replenish the contents of the scratch pad memory. The scratch pad memory is typically implemented as a static random access memory (SRAM) array with low latency and high bandwidth capabilities, while the global memory is typically implemented as a dynamic random access memory (DRAM) array external to the chip containing multiple processor cores. The periodicity of data replenishment of the scratch pad memory is determined by the relationship between the size of the application's working set and the physical capacity of the scratch pad memory. The larger the scratch pad memory, the less frequently it is replenished. Summary of the invention

[0003] A system and method for increasing the capacity of a main on-chip scratch pad memory for a processing core is provided.

[0004] In one aspect, a memory system is provided that is configured to increase the capacity of multiple primary scratch pad memories for multiple corresponding processing cores. The memory system includes a global memory device coupled to multiple processing elements. The global memory device is placed outside of the chip where the multiple processing elements reside. The memory system also includes at least one primary scratch pad memory coupled to a processing element of the multiple processing elements and the global memory device. The memory system also includes multiple auxiliary scratch pad memories coupled to the multiple processing elements and the global memory device. At least a portion of the multiple auxiliary scratch pad memories are configured as a single multi-channel device. Therefore, bandwidth constraints are alleviated by overcoming capacity constraints of existing memory-intensive systems, such as those implemented in neural network designs.

[0005] In another aspect, a method of assembling a memory system is presented that is configured to increase the capacity of multiple primary scratch pad memories for multiple corresponding processing cores. The method includes placing multiple processing elements on a chip and coupling a global memory device to the multiple processing elements. The global memory device is placed external to the chip. The method also includes coupling at least one primary scratch pad memory to at least one of the multiple processing elements and the global memory device. The method also includes coupling multiple auxiliary scratch pad memories to the multiple processing elements and the global memory device. At least a portion of the multiple auxiliary scratch pad memories are configured as a single multi-channel device. Thus, bandwidth constraints are mitigated by overcoming capacity constraints of existing memory-intensive systems, such as those implemented in neural network designs.

[0006] In yet another aspect, a computer system is provided that is configured to increase the capacity of multiple primary scratch pad memories for multiple corresponding processing cores. The computer system includes multiple processing devices placed on a chip. Each of the one or more processing devices includes one or more processing elements. The computer system also includes at least one primary scratch pad memory coupled to the one or more processing elements. The computer system also includes a global memory device coupled to the multiple processing devices. The global memory device is placed outside the chip, and the global memory device is coupled to at least one primary scratch pad memory. The computer system also includes one or more auxiliary scratch pad memories coupled to the multiple processing devices and the global memory device. At least a portion of the multiple auxiliary scratch pad memories are configured as a single multi-channel device. Therefore, bandwidth constraints are alleviated by overcoming capacity constraints of existing memory-intensive systems, such as those implemented in neural network designs.

[0007] The present invention does not intend to show every aspect and / or every embodiment of each embodiment of the present disclosure. These and other features and advantages will become apparent from the following detailed description of the present embodiment in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings included in this application are incorporated into the specification and form a part of the specification. They show embodiments of the present disclosure and are used to explain the principles of the present disclosure together with the specification. The accompanying drawings are illustrations of certain embodiments rather than limiting the present disclosure.

[0009] Figure 1A is a schematic block diagram illustrating a memory system architecture including multi-channel direct random access memory (DRAM) features according to some embodiments of the present disclosure.

[0010] Figure 1B is a diagram showing some embodiments of the present disclosure Figure 1AA schematic block diagram of an enlarged view of a portion of the memory system architecture shown in .

[0011] Figure 2A is a schematic block diagram illustrating a memory system architecture including a multi-channel DRAM feature according to some embodiments of the present disclosure.

[0012] Figure 2B is a diagram showing some embodiments of the present disclosure Figure 2A A schematic block diagram of an enlarged view of a portion of the memory system architecture shown in .

[0013] Figure 3A is a schematic block diagram illustrating a memory system architecture including a multi-channel DRAM feature according to some embodiments of the present disclosure.

[0014] Figure 3B is a diagram showing some embodiments of the present disclosure Figure 3A A schematic block diagram of an enlarged view of a portion of the memory system architecture shown in .

[0015] Figure 4 is a schematic block diagram illustrating a memory system architecture including a multi-channel DRAM feature according to some embodiments of the present disclosure.

[0016] Figure 5 is a schematic block diagram illustrating a memory system architecture including a multi-channel DRAM feature according to some embodiments of the present disclosure.

[0017] Figure 6 is a schematic block diagram illustrating a computing system according to some embodiments of the present disclosure.

[0018] Figure 7 is a flow chart illustrating a process for assembling a computer system according to some embodiments of the present disclosure.

[0019] Although the present disclosure may be modified into various modifications and alternative forms, the details thereof have been shown by way of example in the accompanying drawings and will be described in detail. However, it should be understood that the purpose is not to limit the present disclosure to the specific embodiments described. On the contrary, the present invention will cover all modifications, equivalents and alternatives that fall within the scope of the present disclosure. DETAILED DESCRIPTION

[0020] Aspects of the present disclosure relate to increasing the capacity of a primary on-chip scratch pad memory for a processing core. While the present disclosure is not necessarily limited to such applications, various aspects of the present disclosure may be understood through a discussion of various examples using this context.

[0021] It is readily understood that the components of the present embodiments, as generally described and illustrated in the drawings herein, may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the apparatus, system, method, and computer program product of the present embodiments, as presented in the drawings, is not intended to limit the scope of the embodiments as claimed, but is merely representative of selected embodiments.

[0022] Throughout the specification, references to "selected embodiments," "at least one embodiment," "one embodiment," "another embodiment," "other embodiments," or "embodiments," and similar language mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the phrases "selected embodiments," "at least one embodiment," "in one embodiment," "another embodiment," "other embodiments," or "embodiments" appearing in various places in this specification are not necessarily referring to the same embodiment.

[0023] The illustrated embodiments will be better understood by referring to the drawings, wherein like components are always represented by like reference numerals.The following description is intended to be exemplary only and only illustrates certain selected embodiments of devices, systems, and processes consistent with the embodiments claimed herein.

[0024] As used herein, "facilitating" an action includes performing the action, making the action easier, helping to perform the action, or causing the action to be performed. Thus, by way of example and not limitation, instructions executed on a processor may facilitate an action performed by a semiconductor processing device by sending appropriate data or commands to cause or help perform the action. Where an actor facilitates an action by performing an action other than the action, the action is still performed by some entity or combination of entities.

[0025] Typically, as the robustness of modern computing systems increases, additional memory strategies are implemented to reduce access times, where performance is limited by the speed and efficiency of operations using the memory bus. The use of a memory controller reduces access times to some extent, but generally cannot overcome all limitations of a memory bus that is relatively far away from a processing device. Simply increasing the amount of accessible memory does not alleviate the extended access time. In addition, defining multiple separate channels between the memory and the processing device tends to reduce latency; however, using software features to manage data flow through various channels may tend to increase latency back. Other known solutions include using three-dimensional stacked DRAM (3D DRAM) to localize communications between memory element columns and their underlying processor cores to improve access times. However, some of these solutions use multiplexing devices, which increase access times for certain operations to some extent.

[0026] More specifically, at least some known specialized accelerators, such as but not limited to deep learning accelerators, include a set of interconnected processing cores within a multi-core processing device, where each core uses local memory or scratch pad memory to quickly access its working set, and can access a global memory device to periodically or continuously replenish the contents of the scratch pad memory. The working set is the memory required by a process in a given time interval. For example, with respect to the working set size, for a 100-gigabyte (GB) main memory system, only 50 megabytes (MB) may be necessary for each second of operation of a particular application; therefore, for this case, 50MB is the working set size.

[0027] Scratchpad memories are typically implemented as static random access memory (SRAM) arrays with low latency and high bandwidth capabilities placed directly on the corresponding chip or processing core, where the close proximity of the main scratchpad memory and the processing elements promotes the low latency aspect of the on-chip device. In contrast, global memory is typically implemented as a dynamic random access memory (DRAM) array external to the chip containing multiple processor cores. The periodicity of data replenishment of the scratchpad memory is determined by the relationship between the working set size of the application and the physical capacity of the scratchpad memory. In general, scratchpad memories are distinguished from memory caches, where scratchpad memories tend to be more deterministic about the content of the data residing thereon, thereby facilitating fetching and prefetching operations on only certain data. Therefore, for certain processing requirements, scratchpad memories are superior to caches.

[0028] At least some known efforts to reduce these overheads and improve the utilization of processing resources typically include creating a memory hierarchy in which additional memory layers are inserted directly between the main scratch pad memory and the global memory device. These layers are typically implemented as caches of the main global memory, and are supplemented by moving content from one memory layer to the next memory layer in succession. When the movement between layers is implemented in hardware, special circuits are provided to determine which positions in each layer are to be moved, and when they should be moved, to maximize the utilization of processing resources. Additional hardware is used to implement the layers of memory on the same chip as the processor, and to control the transmission between the layers, and the hardware may be quite complex. In addition, the additional hardware components are usually unaware of the intention of the program executed on the processor, and therefore cannot maximize the processor utilization. They also consume valuable space on the processor chip, which otherwise may have been used to increase the processing power of the chip.

[0029] In addition, some known efforts to reduce these overheads and improve the utilization of processing resources typically include increasing the size of the main scratch pad. Generally, the larger the main scratch pad, the less frequent data replenishment is needed. More frequent replenishment promotes the increase in the processing overhead in terms of the power of transmitting data back and forth and the time required for these transmissions. In addition, the time required for such transmissions always leads to the idleness of processing resources and therefore leads to underutilization of the processor. However, simply increasing the size of the main scratch pad is often inconsistent with the efforts to reduce the size of the chip on which the main scratch pad resides. Therefore, the opportunity cost of implementing an additional layer on the chip as described above is also a similar disadvantage of simply increasing the size of the local scratch pad. In addition, increasing the size of the main scratch pad also increases the delay of accessing the main scratch pad, even for applications with smaller working sets that may not require larger scratch pad resources.

[0030] Therefore, there is a need to alleviate bandwidth constraints through devices that overcome the capacity constraints of existing memory-intensive systems, such as those implemented in neural network designs.

[0031] refer to Figure 1A , presents a schematic block diagram illustrating a memory system architecture 100 (sometimes referred to herein as memory system 100) including multi-channel direct random access memory (DRAM) features according to some embodiments of the present disclosure. In some embodiments, memory system architecture 100 is integrated on a single die. In some embodiments, at least a portion of memory system architecture 100 includes at least a portion of neural network accelerator chip 101; however, memory system architecture 100 is not limited to this embodiment and is configured for use in any embodiment and device architecture that implements the operation of memory system architecture 100 as described herein.

[0032] In some embodiments, memory system architecture 100 includes global memory device 102. In at least some embodiments, global memory device 102 is implemented as a dynamic random access memory (DRAM) device. In some embodiments, global memory device 102 is implemented as a DRAM array. In some embodiments, global memory device 102 is on the same die as neural network accelerator chip 101. In some embodiments, global memory device 102 is integrated into a chip separate from neural network accelerator chip 101 (e.g., Figure 1A shown).

[0033] In one or more embodiments, the neural network accelerator chip 101 includes a memory controller 104 that is communicatively and operably coupled to the global memory device 102. The memory controller 104 is a device that manages the flow of data sent to and from the global memory device 102. In at least some embodiments, the neural network accelerator chip 101 includes an on-chip interconnect 106 that is communicatively and operably coupled to the memory controller 104 through an interconnect conduit 108. In addition, in some embodiments, the neural network accelerator chip 101 includes a plurality of processing cores 110 from processing core 110-1 to 110-n, where processing core 110-n-1 is read as the “nth minus 1” processing core, and so on, and where the variable “n” has any value that enables the neural network accelerator chip 101 to operate, including but not limited to values ​​of 1 to 16. Each processing core 110 is connected to the memory controller 104 through an on-chip interconnect channel 112 (shown in dashed lines and for clarity in FIG. 1 ). Figure 1A 106. Processing core 110 will refer to Figure 1B Further discussion.

[0034] In addition, the neural network accelerator chip 101 includes a multi-channel memory array 120, referred to herein as a multi-channel auxiliary DRAM array 120. The multi-channel auxiliary DRAM array 120 is configured to augment the global memory device 102. In some embodiments, the multi-channel auxiliary DRAM array 120 is a single device (e.g., Figure 1A ); however, in some embodiments, the multi-channel auxiliary DRAM array 120 is distributed into multiple auxiliary DRAM devices to allow other chip architecture elements to be most appropriately positioned. The multi-channel auxiliary DRAM array 120 includes multiple auxiliary DRAM elements or channels, referred to herein as auxiliary scratch pad memories 122 from auxiliary scratch pad memories 122-1 to auxiliary scratch pad memories 122-n. Therefore, the neural network accelerator chip 101 includes multiple multilateral segments in the form of auxiliary scratch pad memories 122, wherein at least a portion of the multiple auxiliary scratch pad memories 122 are configured as a single multi-channel device, namely one or more multi-channel auxiliary DRAM arrays 120.

[0035] In some embodiments, each auxiliary scratch pad memory 122 is directly, communicatively, and operatively coupled to a respective processing core 110 via an auxiliary scratch pad memory channel 124 (only two are shown and labeled for clarity, namely, 124-1 and 124-n), e.g., auxiliary scratch pad memory 122-1 is coupled to processing core 110-1, and auxiliary scratch pad memory 122-n is coupled to processing core 110-n. In some embodiments, as opposed to a one-to-one relationship between auxiliary scratch pad memories 122 and processing cores 110, at least some processing cores 110 are coupled to multiple auxiliary scratch pad memories 122. In some embodiments, a respective auxiliary scratch pad memory channel 124 cooperates with an associated auxiliary scratch pad memory 122 to define a respective auxiliary DRAM channel. In some embodiments, global memory device 102 and multi-channel auxiliary DRAM array 120 are physically implemented side-by-side with each other in a parallel configuration. Thus, the multi-channel auxiliary DRAM array 120 and the respective auxiliary scratch pad memories 122 are located outside of the respective processing core (or chip) 110, thereby allowing the designer of the processing core chip 110 to use the unused portions of the processing core 110 for other features. In addition, the use of DRAM for the auxiliary scratch pad memory 122 helps to take advantage of the density of DRAM technology compared to less dense SRAM or embedded DRAM technology. In some embodiments, such use of DRAM technology allows the auxiliary scratch pad memory 122 to contain the working set of, for example, but not limited to, a large deep learning inference application targeting natural language understanding.

[0036] refer to Figure 1B , presents a schematic block diagram showing an enlarged view of a portion 150 of a memory system architecture 100 according to some embodiments of the present disclosure, and more specifically, a neural network accelerator chip 101 ( Figure 1A An enlarged view of a portion of the Figure 1A . In one or more embodiments, each processing core 110 includes a main on-chip scratch pad memory 152, referred to herein as a main scratch pad memory 152, coupled to the on-chip interconnect 106 through a corresponding on-chip interconnect channel 112. In some embodiments, the main scratch pad memory 152 at least partially defines the on-chip interconnect channel 112. In some embodiments, the main scratch pad memory 152 is configured as a static random access memory (SRAM) array with low latency and high bandwidth capabilities placed directly on the corresponding chip or processing core 110, wherein the close proximity of the main scratch pad memory 152 and the processing element 158 ​​facilitates the low latency aspect of the on-chip device of the memory system architecture 100. In some embodiments, the main scratch pad memory 152 has any memory architecture that enables operation of the memory system architecture 100 including the neural network accelerator chip 101, including but not limited to DRAM devices.

[0037] Furthermore, in some embodiments, each processing core 110 includes a channel controller 154 coupled to a corresponding auxiliary scratch pad memory 122 via a corresponding auxiliary scratch pad memory channel 124. The channel controller 154 is configured to manage the transmission of signals from the corresponding auxiliary scratch pad memory 122 to the processing element 158. In some embodiments, the channel controller 154 at least partially defines the auxiliary scratch pad memory channel 124. Furthermore, in some embodiments, each processing core 110 includes a multiplexer (MUX) 156 configured to direct selected signals from the corresponding primary scratch pad memory 152 and the corresponding auxiliary scratch pad memory 122 (via the channel controller 154) to the processing element 158 ​​for the desired processing operation. Thus, the corresponding auxiliary scratch pad memory 122 is physically placed off-chip with the processing core 110 and is mapped to the corresponding processing element 158 ​​in parallel with the corresponding primary scratch pad memory 152.

[0038] In some embodiments, by implementing the memory characteristics of auxiliary scratch pad 122 to augment the memory characteristics of primary scratch pad 152, the contents of the memory and the orchestration of the movement of the contents to and from the respective processor cores 110 can be orchestrated entirely in software, thereby allowing utilization of processing resources to approach a theoretical maximum. In some embodiments, due to the improved performance of neural network accelerator chip 101 as a result of the enlargement of the memory system as described herein, a reduction in the overall energy consumption of the processing cores 110 can be achieved.

[0039] In at least some embodiments, auxiliary scratch pad 122 is configured to facilitate processing of static and dynamic tensors within neural networks, such as those found within deep learning platforms, including but not limited to, at least in part, Figure 1A and 1B. In addition, the auxiliary scratch pad memory described herein is applicable to a wider range of artificial intelligence platforms, including but not limited to machine learning platforms. In general, a tensor represents a multidimensional array containing elements of a single data type configured for arbitrary numerical calculations. Therefore, in at least some embodiments, the deep learning tensor is configured as one or more matrices represented using an n-dimensional array (wherein the term "n-dimensional" used for the tensor is not associated with the variable number of processing cores 110 and auxiliary scratch pad memories 122 described herein). In a non-limiting example, numerical data in the form of 27 values ​​is stored in a single array, which is implemented as a single continuous block in memory as a 3 by 3 by 3 tensor, one value follows another value, where the three dimensions are also non-limiting. In some embodiments, the tensor is static in shape, that is, the number of dimensions in the matrix and the range of each dimension do not change over time. A non-limiting example of the value of a static tensor includes a model weight, which is a read-only value and is adopted during the definition of the corresponding computational graph of the node corresponding to the operation or variable therein in the neural network (sometimes referred to as reasoning time, where the model remains fixed). In contrast, in some embodiments, tensors are dynamic, i.e., their values ​​are not fixed, and they are generated as intermediate or final outputs, where their lifetime is not necessary for full execution of the program. In some embodiments, for example but not limited to, during the execution of a corresponding computation graph, the matrix configuration with respect to the number of dimensions and the range of each dimension within the array is subject to change.

[0040] In at least some embodiments, the auxiliary scratch pad 122 is configured to store static deep learning tensors, including, but not limited to, weighted values ​​used to facilitate the operation of the corresponding neural network. In contrast, the primary scratch pad 152 is configured to hold dynamic tensors that are also used to facilitate the operation of the corresponding neural network, such as, but not limited to, node activations. In some embodiments, the auxiliary scratch pad 122 is configured with a read bandwidth that is lower than the read bandwidth of the primary scratch pad 152, wherein the more limited static content of the auxiliary scratch pad 122 does not require a greater read bandwidth that is typically found in the primary scratch pad 152. In addition, due at least in part to the above-described properties of the data stored in the auxiliary scratch pad 122, the write bandwidth of the auxiliary scratch pad 122 is lower than the read bandwidth of the auxiliary scratch pad 122. Therefore, in such embodiments, the read / write bandwidth of the auxiliary scratch pad 122 does not need to be as large as the read / write bandwidth of the primary scratch pad 152.

[0041] In some embodiments, the read / write bandwidth of the auxiliary scratch pad 122 is expanded to accommodate storing dynamic tensors, which is necessary for those embodiments of the memory system 100 and neural network accelerator chip 101 that require such a configuration.

[0042] In one or more embodiments, the neural network accelerator chip 101 includes a one-to-one relationship between auxiliary scratch pad memories 122 and processing cores 110. In some embodiments, the neural network accelerator chip 101 includes more than a one-to-one relationship between auxiliary scratch pad memories 122 and processing cores 110, where one limiting factor includes the amount of remaining space availability relative to the physical landscape of the neural network accelerator chip 101. Therefore, in some embodiments, multiple auxiliary scratch pad memories 122 are coupled to corresponding MUXs 156 through one of the separate corresponding channel controllers 154 or a single multi-channel controller (not shown). In some embodiments, the processing element 158 ​​is configured to accommodate any number of auxiliary scratch pad memories 122, including but not limited to 1 to 16 auxiliary scratch pad memories 122.

[0043] refer to Figure 2A , presents a schematic block diagram illustrating a memory system architecture 200 (sometimes referred to herein as memory system 200) including multi-channel DRAM features according to some embodiments of the present disclosure. In many embodiments, memory system 200 is similar to memory system 100 ( Figure 1A and 1B ), one difference being the addition of global channels 230 (only one is labeled). Figure 2B , presents a diagram showing some embodiments according to the present disclosure Figure 2A 2 is a schematic block diagram of an enlarged view of a portion 250 of the memory system architecture 200 shown in FIG. Figure 1A , 1B , 2A and 2B are similarly named with similar functionality.

[0044] Global channels 230 are communicatively coupled to on-chip interconnect 206, which is communicatively coupled to global memory device 202. Additionally, global memory channels 230 are communicatively and operatively coupled to each of processing cores 210, thereby directly coupling global memory device 202 to processing cores 210. This direct coupling facilitates those situations where there are data calls by the respective processing elements 258 to data residing in other portions of memory system 200.

[0045] In at least some embodiments, global channels 230 are coupled to channel controller 254 to facilitate managing the flow of information into and out of MUX 256 for processing elements 258. In some embodiments, global channels 230 are coupled to processing elements 258 by any mechanism that enables memory system 200 to function as described herein.

[0046] refer to Figure 3A , presents a schematic block diagram illustrating a memory system architecture 300 (sometimes referred to herein as memory system 300 ) including multi-channel DRAM features according to some embodiments of the present disclosure. Figure 1A , 1B , 2A and 2B are similarly named with similar functionality and continue to refer to Figure 1A , 1B , 2A and 2B.

[0047] In at least some embodiments, at least a portion of memory system 300 includes one or more neural network accelerator chips in a manner similar to network accelerator chips 101 / 201. In some embodiments, memory system 300 includes an on-chip interconnect 306 that is substantially similar to on-chip interconnect 106 / 206. In at least some embodiments, memory system 300 includes a global memory device (which is substantially similar to global memory device 102 / 202) that is communicatively coupled to on-chip interconnect 306 by a memory controller (which is substantially similar to memory controller 104 / 204) via an interconnect conduit 308 (which is substantially similar to interconnect conduit 108 / 208), where for clarity, the on-chip interconnect 306 is shown in FIG. Figure 3A The global memory devices and memory controllers are not shown.

[0048] In one or more embodiments, the memory system 300 includes a plurality of chiplets 340, i.e., a first chiplet 340-1, a second chiplet 340-2, and so on to an mth chiplet 340-m, wherein the variable "m" has any value that enables the memory system 300 to operate, including but not limited to values ​​1 to 16. In some embodiments, the chiplets 340 are located in a neural network accelerator chip ( Figure 3A Each chiplet 340 includes one or more processing cores 310, where, for example, Figure 3A1-2. A non-limiting number of two processing cores 310-1-1 and 301-1-2 are shown for the first chiplet 340-1. Similarly, the second chiplet 340-2 includes, for example, a non-limiting number of two processing cores 310-2-1 and 301-2-2, and the mth chiplet 340-m includes, for example, a non-limiting number of two processing cores 310-m-1 and 301-m-2. In some embodiments, each chiplet 340 includes any number of processing cores 310 capable of implementing the operations of the memory system 300 as described herein, including but not limited to 1 to 16 processing cores 310. Each processing core 310 is interconnected via an on-chip interconnect channel 312 (for clarity, in FIG. Figure 3A 306. Processing core 310 will refer to Figure 3B Further discussion.

[0049] Additionally, in at least some embodiments, each chiplet 340 includes one or more auxiliary DRAM elements or channels, referred to herein as auxiliary scratchpad memory 322. Figure 3A As shown, each chiplet 340 includes one or more auxiliary scratch pad memories 322, wherein, for example, Figure 3A 3. A non-limiting number of two auxiliary scratch pad memories 322-1-1 and 322-1-2 are shown for the first chiplet 340-1. Similarly, the second chiplet 340-2 includes, for example, a non-limiting number of two auxiliary scratch pad memories 322-2-1 and 322-2-2, and the mth chiplet 340-m includes, for example, a non-limiting number of two auxiliary scratch pad memories 322-m-1 and 322-m-2. In some embodiments, each chiplet 340 includes any number of auxiliary scratch pad memories 322 that can implement the operation of the memory system 300 as described herein, including but not limited to 1 to 16 auxiliary scratch pad memories 322.

[0050] In some embodiments, the plurality of auxiliary scratch pad memories 322 for each chiplet 340 are separate components placed on each chiplet 340, such that each chiplet 340 is a single device. Conversely, in some embodiments, the plurality of chiplets 340 are formed as a single multi-channel auxiliary DRAM array 320 (e.g., Figure 3A Thus, in such an embodiment, the multi-channel auxiliary DRAM array 320 includes a plurality of multi-sided segments in the form of auxiliary scratch pad memories 322, wherein at least a portion of the plurality of auxiliary scratch pad memories 322 are configured as a single multi-channel device, e.g., each chiplet 340 and the multi-channel auxiliary DRAM array 320 is such a device.

[0051] In some embodiments, each auxiliary scratch pad memory 322 is directly, communicatively, and operatively coupled to a corresponding processing core 310 via an auxiliary scratch pad memory channel 324 (only one is shown and labeled for clarity), e.g., auxiliary scratch pad memory 322-1-1 is coupled to processing core 310-1-1, and auxiliary scratch pad memory 322-m-2 is coupled to processing core 310-m-2. In some embodiments, as opposed to a one-to-one relationship between auxiliary scratch pad memories 322 and processing cores 310, at least some processing cores 310 are coupled to multiple auxiliary scratch pad memories 322. In some embodiments, a respective auxiliary scratch pad memory channel 324 cooperates with an associated auxiliary scratch pad memory 322 to define a respective auxiliary DRAM channel.

[0052] See also Figure 3B , presents a diagram showing some embodiments according to the present disclosure Figure 3A 3 is a block schematic diagram of an enlarged view of a portion 350 of the memory system architecture 300 shown in . In many embodiments, portion 350 of memory system 300 is substantially similar to portion 250 of memory system 200 . Figure 2B and 3B Like numbered components in the are similarly named with similar functionality.

[0053] See also Figure 4 , presents a schematic block diagram illustrating a memory system architecture 400 including multi-channel DRAM features according to some embodiments of the present disclosure. Also refer to Figure 1A , 1B , 2A, 2B, 3A, and 3B, in at least some embodiments, the memory system architecture 400 includes at least one logical processor die 402, where the number one is non-limiting. In some embodiments, the logical processor die 402 includes a plurality of processing cores 410 substantially similar to the processing cores 110, 210, and 310. The logical processor die 402 includes any number of processing cores 410 that enables the memory systems 100, 200, 300, and 400 to implement the operations described herein, including but not limited to 1 to 16, where Figure 4 16 units of processing cores 410 are shown in FIG. 4 , and eight processing cores 410 are labeled from processing core 410 - 1 to 410 - 8 .

[0054] Additionally, in some embodiments, the memory system architecture 400 includes a plurality of DRAM die 420. In some embodiments, the DRAM die 420 is similar to the multi-channel auxiliary DRAM array 120, 220, or 320. The memory system architecture 400 includes any number of DRAM die 420 that enables the memory systems 100, 200, 300, and 400 to operate as described herein, including but not limited to 1 to 16, wherein the DRAM die 420 is substantially the same as the multi-channel auxiliary DRAM array 120, 220, or 320. Figure 4 4 units of DRAM die 420 are shown, namely DRAM die 420-1, 420-1, 420-3 and 420-4. As shown, the memory system architecture 400 defines a three-dimensional (3D) stacked configuration. In some embodiments, the number of stacked DRAM die 420 is subject to environmental conditions, including but not limited to thermal considerations regarding heat generation and removal. In addition, the number of selected stacked DRAM die 420 is subject to chip design considerations regarding the positioning of various components thereon, as well as practical limitations based on modern chip manufacturing technology and the structural strength requirements of the DRAM die 420.

[0055] In one or more embodiments, each DRAM die 420 includes a plurality of auxiliary scratch pad memories 422 that are substantially similar to auxiliary scratch pad memories 122. In some embodiments, the plurality of auxiliary scratch pad memories 422 are substantially similar to auxiliary scratch pad memories 222 and 322. In some embodiments, each DRAM die 420 includes any number of auxiliary scratch pad memories 422 that enables memory systems 100, 200, 300, and 400 to operate as described herein, including but not limited to 1 to 16, wherein at least one auxiliary scratch pad memory 422 is substantially similar to auxiliary scratch pad memories 222 and 322. Figure 4 16 cells of auxiliary scratch pad memory 422 are shown for each of the four DRAM die 420, and eight processing cores 410 are labeled from processing core 410-1 to 410-8. In some embodiments, electrical connections through the stack are facilitated by a plurality of through silicon vias (TSVs) 430. Communication between the processing cores 410 and the auxiliary scratch pad memory 422 is referenced to FIG. Figure 1A , 1B , 2A, 2B, 3A and 3B for discussion.

[0056] like Figure 4 As shown, in some embodiments, each processor core 410 is communicatively and operably coupled to four auxiliary scratch pad memories 422. For example, processor core 410-1 is coupled to auxiliary scratch pad memories 422-1-1, 422-2-1, 422-3-1, and 422-4-1. In some embodiments, processor core 410 is coupled to any number of auxiliary scratch pad memories 422 that enables memory system architecture 400 to implement operations as described herein.

[0057] refer to Figure 5 , presents a schematic block diagram illustrating a memory system architecture 500 including a multi-channel DRAM feature according to some embodiments of the present disclosure. The memory system architecture 500 is similar to the memory system architecture 400 ( Figure 4, also continued reference); however, the memory system architecture 500 is not a stacked configuration, but a parallel configuration. Therefore, the memory system architecture 500 includes at least one logical processor die 502 similar to the logical processor die 402. In addition, in some embodiments, the memory system architecture 500 includes one or more DRAM die 520, of which two DRAM die 520-1 and 520-2 are shown. The DRAM die 520 is similar to the DRAM die 420. The communication between the processing core 510 and the auxiliary scratch pad memory 522 is referenced Figure 1A , 1B , 2A, 2B, 3A and 3B for discussion.

[0058] Reference now Figure 6 , provides a schematic block diagram of a computing system 601 that can be used to implement one or more of the methods, tools, and modules described herein and any related functionality (e.g., using one or more processor circuits of a computer or a computer processor) according to some embodiments of the present disclosure. In some embodiments, the main components of the computer system 601 may include one or more CPUs 602, a memory subsystem 604, a terminal interface 612, a storage interface 616, an I / O (input / output) device interface 614, and a network interface 618, all of which may be directly or indirectly communicatively coupled for inter-component communication via a memory bus 603, an I / O bus 608, and an I / O bus interface unit 610.

[0059] Computer system 601 may include one or more general purpose programmable central processing units (CPUs) 602-1, 602-2, 602-3, 602-N, collectively referred to herein as CPUs 602. In some embodiments, computer system 601 may include multiple processors typical of relatively large systems; however, in other embodiments, computer system 601 may instead be a single CPU system. Each CPU 602 may execute instructions stored in a memory subsystem 604 and may include one or more levels of onboard cache.

[0060] System memory 604 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 622 or cache memory 624. Computer system 601 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 626 may be provided for reading from and writing to non-removable, non-volatile magnetic media such as "hard drives". Although not shown, a disk drive for reading from and writing to a removable, non-volatile disk (e.g., "floppy disk") may be provided, or an optical drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media may be provided. In addition, memory 604 may include flash memory, such as a flash stick drive or a flash drive. In addition, global memory devices, primary scratch pad memory, and auxiliary scratch pad memory as described herein are included as part of the described memory device suite. Memory devices may be connected to memory bus 603 via one or more data medium interfaces. The memory 604 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of different embodiments.

[0061] Although the memory bus 603 is Figure 6 603 may include multiple different buses or communication paths, which may be arranged in any of a variety of forms, such as point-to-point links in a hierarchical star or mesh configuration, multi-layer buses, parallel and redundant paths, or any other suitable type of configuration. In addition, although the I / O bus interface 610 and the I / O bus 608 are shown as a single corresponding unit, in some embodiments, the computer system 601 may include multiple I / O bus interface units 610, multiple I / O buses 608, or both. In addition, although multiple I / O interface units are shown to separate the I / O bus 608 from the various communication paths extending to various I / O devices, in other embodiments, some or all of the I / O devices may be directly connected to one or more system I / O buses.

[0062] In some embodiments, computer system 601 may be a multi-user mainframe computer system, a single-user system, or a server computer or similar device with little or no direct user interface but receiving requests from other computer systems (clients). In addition, in some embodiments, computer system 601 may be implemented as a desktop computer, a portable computer, a laptop or notebook computer, a tablet computer, a pocket computer, a phone, a smart phone, a network switch or router, or any other suitable type of electronic device.

[0063] Notice, Figure 6 The main components of the exemplary computer system 601 are intended to be depicted. However, in some embodiments, the various components may have more Figure 6 The greater or lesser complexity represented in , may exist different from Figure 6 Components may be those shown in or in addition to those shown in, and the number, type, and configuration of these components may vary.

[0064] One or more programs / utilities 628 may be stored in memory 604, each having at least one set of program modules 630. Programs / utilities 628 may include a hypervisor (also referred to as a virtual machine monitor), one or more operating systems, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. Programs 628 and / or program modules 630 generally perform the functions or methods of various embodiments.

[0065] refer to Figure 7 A, provides a diagram for assembling a computer system such as 600 (see Figure 6 ) is a flowchart of a process 700 of a computer system. Figure 1A , 1B, 2A, 2B, 3A, 3B, 4, 5 and 6, process 700 includes placing 702 a plurality of processing elements 158 / 258 / 358 on chip 101 / 201. Process 700 also includes coupling 704 a global memory device 102 / 202 to the plurality of processing elements 158 / 258 / 358, wherein the global memory device 102 / 202 is located external to chip 101 / 201. Process 700 also includes coupling 706 at least one primary scratch pad memory 152 / 252 / 352 to at least one processing element of the plurality of processing elements 158 / 258 / 358 and the global memory device 102 / 202. Process 700 also includes coupling 708 a plurality of auxiliary scratch pad memories 122 / 222 / 322 to respective processing elements 158 / 258 / 358 and the global memory device 102 / 202. At least a portion of the plurality of auxiliary scratch pad memories 122 / 222 / 322 are configured as a single multi-channel device, namely any one of the multi-channel auxiliary DRAM arrays 120 / 230 / 330 and the chiplet 340 .

[0066] The process 700 also includes coupling 710 each auxiliary scratch pad memory 122 / 222 / 322 to a corresponding processing core 110 / 210 / 310, thereby defining 712 a plurality of auxiliary scratch pad memory channels 124 / 224 / 324. In some embodiments, coupling a channel controller 154 / 254 / 354 to a corresponding auxiliary scratch pad memory 122 / 222 / 322 further defines 712 the corresponding auxiliary scratch pad memory channels 124 / 224 / 324. In some embodiments, a neural network accelerator chip 101 / 201 is fabricated 714. In some embodiments, a plurality of chiplets 340 are fabricated 716. Both the neural network accelerator chip 101 / 201 and the chiplet 340 include a primary scratch pad memory 152 / 252 / 352, a processing element 158 / 258 / 358, and an auxiliary scratch pad memory 122 / 222 / 322.

[0067] The embodiments disclosed and described herein are configured to provide improvements to computer technology. Materials, operable structures, and techniques as disclosed herein may provide substantially beneficial technical effects. Some embodiments may not have all of these potential advantages, and these potential advantages are not required for all embodiments. By way of example only and not limitation, one or more embodiments may provide enhancements to the operation of a memory system by adding dedicated auxiliary scratch pad memories to each processor core.

[0068] In at least some embodiments as described herein, the enhancement of the memory system includes higher bandwidth, because the off-chip memory bandwidth is generally limited by the corresponding packaging characteristics. In addition, as the computational performance of modern computing systems increases, the memory bandwidth also increases in proportion to the computational gain of balancing system performance. In addition, compared with on-chip memory access, the use of off-chip memory access is relatively power-intensive. Therefore, the embodiments described herein, including 3D stacked embodiments, in which the processor core is closely adjacent to the auxiliary DRAM note memory on the same chip, promote the reduction of power consumption of the associated computer system. In addition, the 3D stacked DRAM embodiments described herein promote greater memory density than SRAM on a bare die; therefore, the embodiments described herein promote much larger memory capacity with a smaller form factor, thereby promoting larger memory capacity. In addition, with respect to form factor, the embodiments described herein facilitate reducing the reliance on memory packaging outside the chip, thereby reducing and in some cases eliminating the need for external memory packaging.

[0069] In one or more embodiments described herein, additional benefits are obtained when smaller batch cases are executed more quickly by the respective computing systems for those operations where amortization of costs associated with importing various weighted values ​​is not feasible, i.e., when such costs are not recoverable.

[0070] The present disclosure may be a system, method and / or computer program product at any possible level of technical detail integration. A computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform aspects of the present disclosure.

[0071] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0072] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0073] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0074] The computer-readable program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc., and procedural programming languages, such as "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider via the Internet). In some embodiments, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions, so as to perform aspects of the present disclosure.

[0075] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart and / or block diagram and the combination of blocks in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0076] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device and / or other equipment to work in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0077] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0078] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure.In this regard, each frame in the flow chart or block diagram can represent the part of module, segment or instruction, which includes one or more executable instructions for realizing the specified logical function.In some alternative embodiments, the function noted in the frame may not occur in the order noted in the figure.For example, the two frames shown continuously can actually be realized as a step, and are executed simultaneously, substantially simultaneously, in a partially or entirely overlapping manner, or these frames can sometimes be executed in reverse order, depending on the function involved.It will also be noted that the combination of the frames in each frame illustrated in the block diagram and / or flow chart and the block diagram and / or flow chart can be realized by a dedicated hardware-based system that performs a specified function or action or performs a combination of special-purpose hardware and computer instructions.

[0079] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements over existing technologies on the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A memory system configured to increase the capacity of a plurality of primary scratch pad memories for a plurality of corresponding processing cores, the memory system comprising: a global memory device coupled to a plurality of processing elements, wherein the global memory device is disposed external to a chip in which the plurality of processing elements reside; at least one master scratch pad memory coupled to at least one of the plurality of processing elements and to the global memory device; as well as A plurality of auxiliary scratch pad memories are coupled to the plurality of processing elements and the global memory device, wherein at least a portion of the plurality of auxiliary scratch pad memories are configured as a single multi-channel device.

2. The memory system of claim 1, wherein: The plurality of secondary scratch pad memories are configured as dynamic random access memory (DRAM).

3. The memory system of claim 2, wherein: The plurality of auxiliary scratch pad memories define a multi-channel auxiliary DRAM array.

4. The memory system of claim 3, wherein: The plurality of processing elements comprises a plurality of processing cores; Each auxiliary scratch pad memory of the plurality of auxiliary scratch pad memories is coupled to a corresponding processing core of the plurality of processing cores, thereby defining a plurality of auxiliary scratch pad memory channels.

5. The memory system according to claim 4, further comprising: A channel controller is coupled to one or more auxiliary scratch pad memories of the plurality of auxiliary scratch pad memories to further define auxiliary scratch pad memory channels of the plurality of auxiliary scratch pad memory channels.

6. The memory system according to claim 1, further comprising: A plurality of chiplets, wherein each chiplet of the plurality of chiplets comprises: said at least one primary scratch pad memory; the at least one processing element; and One or more auxiliary scratch pad memories of the plurality of auxiliary scratch pad memories.

7. The memory system according to claim 1, further comprising: An on-chip interconnect is coupled to the global memory device, at least one processing element of the plurality of processing elements, the at least one primary scratch pad memory, and each of the plurality of auxiliary scratch pad memories.

8. A method for assembling a memory system configured to increase the capacity of a plurality of primary scratch pad memories for a plurality of corresponding processing cores, the method comprising: Placing multiple processing elements on a chip; coupling a global memory device to the plurality of processing elements, wherein the global memory device is disposed external to the chip; coupling at least one master scratch pad memory to at least one processing element of the plurality of processing elements and to the global memory device; as well as A plurality of auxiliary scratch pad memories are coupled to the plurality of processing elements and the global memory device, wherein at least a portion of the plurality of auxiliary scratch pad memories are configured as a single multi-channel device.

9. The method according to claim 8, further comprising: The plurality of auxiliary scratch pad memories are configured as dynamic random access memory (DRAM).

10. The method according to claim 9, further comprising: The plurality of auxiliary scratch pad memories are assembled to define a multi-channel auxiliary DRAM array.

11. The method according to claim 10, further comprising: Each auxiliary scratch pad memory of the plurality of auxiliary scratch pad memories is coupled to a respective processing core of the plurality of processing cores, thereby defining a plurality of auxiliary scratch pad memory channels.

12. The method according to claim 11, further comprising: A channel controller is coupled to one or more auxiliary scratch pad memories of the plurality of auxiliary scratch pad memories to further define auxiliary scratch pad memory channels of the plurality of auxiliary scratch pad memory channels.

13. The method according to claim 8, further comprising: Assembling a plurality of chiplets, wherein each chiplet in the plurality of chiplets comprises: said at least one primary scratch pad memory; the at least one processing element; and One or more auxiliary scratch pad memories of the plurality of auxiliary scratch pad memories.

14. A computer system comprising: a plurality of processing devices disposed on a chip, each of the plurality of processing devices comprising one or more processing elements and at least one primary scratch pad memory coupled to the one or more processing elements; And, the memory system according to any one of claims 1 to 7.

Images