Stacked memory device with error recovery support

Abstract

Technologies for stacked memory devices with error recovery support are described. One stacked memory device includes a base layer and a plurality of memory layers. The base layer includes a primary interface for regular data, a secondary interface for checkpoint data, and a buffer for temporarily storing the checkpoint data. The buffer is accessible by the plurality of memory layers and the secondary interface. The base layer, in response to a first command, initiates a first transfer of a first block of bits from a memory layer of the plurality of memory layers into the buffer. The base layer, in response to a second command, initiates a second transfer of a second block of bits from the buffer out of the stacked memory device via the secondary interface. The second transfer occurs concurrently with a third transfer of data over the primary interface.

Description

Attorney Docket No.: 27170.1090 (L1033PCT)STACKED MEMORY DEVICE WITH ERROR RECOVERY SUPPORTBACKGROUND

[0001] Modern computer systems generally include a data storage device, such as a memory component or device. The memory component may be, for example, a randomaccess memory (RAM) or a dynamic random-access memory (DRAM) device. The memory device includes memory banks made up of memory cells that a memory controller or memory client accesses through a command interface and a data interface within the memory device. The memory devices can be located on a memory module. The memory module can include one or more volatile memory devices.BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The present disclosure is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0003] FIG. l is a block diagram of a computing system with a graphics processing unit (GPU), a central processing unit (CPU), a fabric switch, and multiple High Bandwidth Memories (HBMs) each having a primary interface and a secondary interface according to at least one embodiment.

[0004] FIG. 2 is a block diagram of a base die of an HBM according to at least one embodiment.

[0005] FIG. 3 is a block diagram of a memory die of an HBM according to at least one embodiment.

[0006] FIG. 4 is a flow diagram of a checkpointing process flow according to at least one embodiment.

[0007] FIG. 5 is a block diagram of a stacked memory device with a base layer and multiple memory layers according to at least one embodiment.

[0008] FIG. 6 illustrates an example format of checkpoint data according to at least one embodiment.

[0009] FIG. 7 is a flow diagram of a method of checkpointing in connection with refresh operations according to at least one embodiment.

[0010] FIG. 8 illustrates an example refresh operation that refreshes 4 rows per bank according to at least one embodiment.

[0011] FIG. 9 is a flow diagram of a method of checkpointing in connection with executing a program element according to at least one embodiment.Attorney Docket No.: 27170.1090 (L1033PCT)

[0012] FIG. 10 illustrates a method of operating a stacked memory device with primary and secondary interfaces according to at least one embodiment.

[0013] FIG. 11 illustrates an expansion card with eight GPUs and multiple HBMs coupled to the eight GPUs according to at least one embodiment.

[0014] FIG. 12 illustrates a rack with multiple servers with multiple expansion cards according to at least one embodiment.DETAILED DESCRIPTION

[0015] Technologies for stacked memory devices with error recovery support are described. The following description sets forth numerous specific details, such as examples of specific systems, components, methods, and so forth, in order to provide a good understanding of several embodiments of the present disclosure. It will be apparent to one skilled in the art, however, that at least some embodiments of the present disclosure may be practiced without these specific details. In other instances, well- known components or methods are not described in detail or presented in simple block diagram format to avoid obscuring the present disclosure unnecessarily. Thus, the specific details set forth are merely exemplary. Particular implementations may vary from these exemplary details and still be contemplated to be within the scope of the present disclosure.

[0016] In the context of memory, Reliability, Availability, and Serviceability (RAS) support is important to ensure the reliability, uptime, and maintainability of memory components like DRAM, static RAM (SRAM), or NAND flash, allowing systems to operate reliably over long periods and can recover quickly in the event of a failure. RAS features can be important in enterprise environments, high-performance computing (HPC), and mission-critical applications where memory failures can lead to data corruption, system crashes, or costly downtime. RAS support, however, can have different priorities for different systems with different runtimes. For example, some systems have short program runtimes, small number of nodes, and the programs run on typical servers (e.g., artificial intelligence (Al) inference or machine learning (ML) inference). The RAS goal for these systems is that the processing jobs run error free. In the rare case of an error in a processing job, the processing job can be run again from the beginning since the runtime is relatively small, as compared to other larger systems that have very long program runtimes (e.g., HPC applications, AI / ML training, etc.). In contrast, larger systems with longer program runtimes and larger number of nodes willAttorney Docket No.: 27170.1090 (L1033PCT) have errors during runtime. The RAS goal for these larger systems is that a recovery of an error in a processing job can be restarted from a checkpoint, instead of from the beginning as done for smaller systems described above. These systems need to support checkpointing and error recovery support.

[0017] Checkpointing is a technique used in computing to save a state of a running program or system at certain points in time, so that it can later resume execution from that point in case of failure, crash, or manual restart. Checkpointing is widely used in HPC systems where jobs can run for days or weeks. In case of a node failure, the computation can resume from the last checkpoint rather than restarting from the beginning. A checkpoint captures the complete or partial state of the program at a specific time. This includes data in memory, program counters, open files, and other relevant information required to resume execution. After a checkpoint is saved, if the program crashes or is interrupted, it can be resumed from the last checkpoint rather than starting from the beginning. This drastically reduces the time and resources required to complete long-running processes.

[0018] Aspects and embodiments of the present disclosure can provide checkpointing and error recovery support to a system using a stacked memory device in a connection architecture with a secondary interface incorporated into a base layer integrated circuit. As described herein, the base layer of a stacked memory device can include a primary interface for transferring regular data, a secondary interface for transferring checkpoint data, and a buffer for temporarily storing the checkpoint data. The buffer can be generally accessible by the secondary interface. The buffer can be accessible by memory layers in the stacked memory device. The base layer can initiate transfer of data from one of the memory layers into the buffer and transfers of data out from the stacked memory device via the secondary interface. In some cases, the transfers to or from the buffer can occur concurrently with transfers of regular data over the primary interface.

[0019] Aspects and embodiments of the present disclosure can provide for checkpointing in the memory devices themselves, providing fault tolerance, efficient resource usage, and load balancing. Using the stacked memory devices with secondary interfaces can allow programs to recover from crashes, power failures, or system issues without losing all progress. The stacked memory devices with secondary interfaces can conserve time and resources as computations do not need to restart from the start after a failure. The checkpointing support provided by the stacked memory devices with secondary interfaces can allow migration of processes to different machines for loadAttorney Docket No.: 27170.1090 (L1033PCT) balancing or hardware upgrades without restarting. Aspects and embodiments of the present disclosure can provide for improvements in performance, flexibility, reduction in system overhead, transparency and integration, etc. Aspects and embodiments of the present disclosure can provide better performance in terms of time and energy. Aspects and embodiments of the present disclosure can provide a scalable solution.

[0020] FIG. 1 is a block diagram of a computing system 100 with a GPU 102, a CPU 104, a fabric switch 106, and multiple stacked memory devices, referred to herein as high bandwidth memory devices (HBMs) 108 each having a primary interface 110 and a secondary interface 112 according to at least one embodiment. The computing system 100 can be an integrated set of components that work together to process, store, and manage data. The computing system 100 can include hardware, software, and networking resources, all designed to perform various computational tasks efficiently. The computing system 100 can include various hardware, such as the GPU 102, including the HBMs 108 coupled to the GPU 102, CPU 104, DDR DRAM 118 (also referred to a system memory) coupled to the CPU, storage 116 (e.g., hard drives, solid- state drives, etc.), input / output devices (e.g., keyboard, mouse, display screen), and networking components (network cards, routers). The computing system 100 can also include a fabric switch 106 coupled between the GPU 102, the CPU 104, the fabric attached DRAM 114 (also referred to as remote DRAM), the storage 116. The computing system 100 can include software, such as programs and / or applications, that run on the CPU 104 and / or GPU 102 to perform specific tasks. The software can include an operating system (OS) to provide essential services for managing hardware resources and facilitating software execution, applications, firmware, etc. Computers within a computing system are often interconnected through networks that enable data exchange, resource sharing, and communication between users and devices. This may involve wired connections like Ethernet, wireless technologies such as Wi-Fi® or Bluetooth®, and internet connectivity for accessing online resources. The computing system 100 can be classified based on its size (e.g., personal computers, servers, supercomputers) and purpose (e.g., general -purpose systems, embedded systems). The computing system 100 can be a computer, a desktop computer, a server, a System on Chip (SoC), a portion of a server, a supercomputer, a mobile device, a phone, a tablet, a set-top box (STB), a media device, a network device, or the like.

[0021] The CPU 104 is the primary component of the computing system 100 a computer that performs many diverse processing tasks. Often referred to as the "brain" ofAttorney Docket No.: 27170.1090 (L1033PCT) the computing system 100, the CPU 104 interprets and executes instructions from the DDR DRAM 118 and carries out basic arithmetic, logical, control, and input / output operations specified by the instructions. The CPU 104 can include one or more cores, which allows for parallel processing and improved multitasking capabilities. The CPU 104 can delegate processing jobs to the GPU 102. The GPU 102 can be primarily used in computing systems for tasks that require high-speed parallel processing, such as graphics rendering, machine learning, and scientific simulations. GPU 102 possesses a highly parallel structure with thousands of cores designed to handle multiple operations simultaneously. This architecture enables the GPU 102 to efficiently process large sets of data in real-time, which is crucial for graphics rendering and complex computational tasks. The GPU 102 can have dedicated memory, including the multiple HBMs 108. The use of HBMs 108 help improve memory access efficiency by storing frequently accessed data closer to the cores for faster retrieval. Besides graphical processing, the GPU 102 can be used for general -purpose computing tasks such as deep learning, scientific simulations, and data analytics due to their high parallelism and computational power. GPUs come in various forms, including integrated graphics chips found within the CPU package or dedicated discrete GPU cards that can be added into a computer system's motherboard via PCI Express (PCIe) slots. As illustrated in FIG. 1, the computing system 100 can include a single CPU 104 and a single GPU 102, but in other embodiments, the computing system 100 can include multiple CPUs and multiple GPUs.

[0022] As illustrated in FIG. 1, the CPU 104 is coupled to the GPU 102 over a fabric switch 106. The GPU 102 includes a fabric interface 122 and the CPU 104 includes a fabric interface 124. The GPU 102 and the CPU 104 can communicate over these fabric interfaces via the fabric switch 106. The GPU 102 includes an HBM interface 120 to communication with the HBM 108 via a primary interface 110. The GPU 102 can include multiple HBM interfaces when there are multiple HBMs 108. In addition to the primary interface 110, the HBM 108 also includes a secondary interface 112 that is coupled to the fabric switch 106. This allows the HBM 108 to be accessible via the primary interface 110 or the secondary interface 112. In other embodiments, the HBM 108 can be directly coupled to the CPU 104. For example, the memory devices used for the DDR DRAM 118 can have the primary interface and secondary interface like the HBM 108 as described in more detail below.

[0023] The HBM 108 is a stacked memory device with a base die (also referred to as a base layer) and a stack of one or more memory dies (also referred to as memory layers orAttorney Docket No.: 27170.1090 (L1033PCT) memory cell layers). The base die has the primary interface 110 and the secondary interface 112, and a buffer (also referred to as memory buffer). The primary interface 110 can be used for regular data (also referred to as mission mode data), whereas the secondary interface 112 can be used for checkpoint data. The buffer can be used to temporarily store the checkpoint data. The buffer can be accessible by the plurality of memory dies. The buffer can be accessible by the secondary interface 112. The base layer can include control logic to manage transfers of data between the primary interface 110, the secondary interface 112, the buffer, and the memory dies in the stack. The control logic on the base die can receive commands via the primary interface 110 or the secondary interface 112. The base die (base layer), in response to a first command, can initiate a first transfer of a first block of bits from one of the memory dies into the buffer. This first command can be used in connection with a checkpointing operation. In at least one embodiment, the size or atom of a checkpoint transfer is one or a few pages (e.g., IkB to 4kB). The base die (base layer), in response to a second command, can initiate a second transfer of a second block of bits from the buffer out of the HBM 108 via the secondary interface 112 (or the primary interface 110 in some embodiments). When the second transfer is done on the secondary interface 112, it can be done concurrently or simultaneously with transfers of data over the primary interface 110.

[0024] As illustrated in FIG. 1, the base die (base layer) of the HBM 108 has a normal HBM interface (referred to as the primary interface 110) and another interface (referred to as the secondary interface 112) that is attached to the fabric switch 106 that is connecting the CPU 104, GPU 102, the fabric attached DRAM 114, and the storage 116. It should be noted that FIG. 1 show one implementation of the HBM 108 having a base die with the primary interface 110, the secondary interface 112, and the buffer. In other embodiments, the HBM 108 can be used in other implementations, such as memory attached to or otherwise associated with a CPU, memory attached to or otherwise associated with a data processing unit (DPU), memory attached to or otherwise associated with other processing components. Additional details of the HBM 108 are illustrated and described below with respect to FIG. 2.

[0025] FIG. 2 is a block diagram of a base die 200 of an HBM according to at least one embodiment. The base die 200 (base layer) includes a buffer 202, a secondary interface 204, a secondary interface control logic and data switch 206, a secondary Through- Silicon Via (TSV) 208, channel logic 210, and channel of primary interface 212. The channel logic 210 can be coupled to n number of channels of the primary interface.Attorney Docket No.: 27170.1090 (L1033PCT)

[0026] As illustrated in FIG. 2, a first channel 212 (channel 0) of the primary interface is coupled to first channel logic 210, which is coupled to a primary TSV for the respective channel. An nth channel 212 (channel n-1) of the primary interface is coupled to nth channel logic 210, which is coupled to a primary TSV for the respective channel. As such, the HBM includes a set of primary TSVs coupled to channels of the primary interface of the base die 200. Each primary TSV can be coupled between the base die 200 and one or more memory dies (memory layers) of the HBM. The HBM can also include a set of one or more secondary TSVs, such as secondary TSV 208 illustrated in FIG. 2, coupled between the base die 200 and one or more memory dies (memory layers) of the HBM. The secondary TSV 208 is coupled to the secondary interface control logic and data switch 206. The secondary TSV 208 can provide an alternate path between the memory dies and the buffer 202. The secondary interface control logic and data switch 206 can also provide a path between the memory dies and the buffer 202 via the primary interface and primary TSVs. The set of secondary TSVs can allow simultaneous transfer or checkpoint data and regular data.

[0027] In at least one embodiment, the secondary interface control logic and data switch 206 can be coupled to the channel logic 210, the set of primary TSVs, the secondary TSV 208, the buffer 202, and the secondary interface 204. A TSV is an advanced semiconductor technology that allows vertical electrical connections to pass through silicon wafers or dies, enabling the stacking of multiple layers of integrated circuits (ICs) or components in a 3D arrangement. TSVs create a direct electrical pathway through the silicon wafer by forming vertical vias (tiny cylindrical holes) that are filled with a conductive material, typically copper. These vias connect different layers of a stacked structure, providing high-speed data transfer between them. TSVs allow for the creation of compact, high-performance multi-layer semiconductor devices.

[0028] The secondary interface control logic and data switch 206, in response to the first command, can initiate the first transfer of the first block of bits from the memory layer into the buffer 202 from the secondary TSV 208. The secondary interface control logic and data switch 206 can receive commands both through the primary interface and the secondary interface 204. Data and commands can be routed between each channel, the buffer 202, and the secondary interface 204.

[0029] In at least one embodiment, the secondary TSV 208 can be avoided if the primary TSVs are used to transfer data from the memory die to the data switch 206 on the base die 200. This would block one channel for the duration of a page transfer everyAttorney Docket No.: 27170.1090 (L1033PCT) time a checkpoint command is issued to that channel. In this embodiment, the secondary interface control logic and data switch 206 can be coupled to the channel logic 210, the secondary interface control logic and data switch 206, in response to the first command, can initiate the first transfer of the first block of bits from the memory layer into the buffer 202 from one of the channel of primary interface 212.

[0030] In at least one embodiment, the buffer can include two or more portions. In at least one embodiment, the buffer includes a first portion and a second portion. The first portion and the second portion can alternate between being written to and being read from. In at least one embodiment, the first data is written to the first portion at a first time, and second data is read from the second portion at the first time. Third data is written to the second portion at a second time, and fourth data is read from the first portion at the second time. This is an example of “double buffering” where the buffer is split into two parts that are accessed in different ways at different times. Other systems can use “triple buffering” with three portions. Triple buffering can be used where it is difficult to keep the two portions in synchronization (i.e., an entity or process that is writing the data can get ahead of another entity or process that is reading the data). Alternatively, the buffer may have more than three portions, more than three portions can

[0031] In at least one embodiment, the first command can be part of a checkpointing operation. The first command can be issued over the primary interface. In at least one embodiment, the secondary interface control logic and data switch 206 can receive the first command over the primary interface. In at least one embodiment, the second command can be part of a recovery operation. The second command can be issued over the secondary interface 204. In other embodiments, the first command and the second command are received over the secondary interface 204. In other embodiments, the secondary interface control logic and data switch 206 can receive commands over any combination of the primary and secondary interfaces.

[0032] In at least one embodiment, the secondary interface 204 is coupled to a fabric interface (e.g., fabric interface 124) of a computing system in which the HBM resides. The fabric interface can operate according to a network standard of the computing system in which the stacked memory device resides. The network standard can be NVLink, InfiniBand, Ethernet, PCIe, or the like. In at least one embodiment, the fabric interface operates independently from a processing unit of the computing system. The processing unit can be a CPU, a GPU, a DPU, or the like.Attorney Docket No.: 27170.1090 (L1033PCT)

[0033] In at least one embodiment, the primary interface can have a first bandwidth, and the secondary interface can have a second bandwidth less than the first bandwidth. In some embodiments, the checkpoint data can be read from the buffer 202 via the primary interface because of the higher bandwidth. In other embodiments, the checkpoint data can be read from the buffer via the secondary interface 204. In at least one embodiment, the checkpoint data can be read in response to a third command. In at least one embodiment, the third command can be part of a recovery operation. The third command can be issued over the secondary interface 204 or the primary interface. In at least one embodiment, the secondary interface control logic and data switch 206 can receive the third command. The secondary interface control logic and data switch 206 can read a stored checkpoint stored in the buffer 202 in response to the third command. The stored checkpoint can be read via the secondary interface 204 (or the primary interface).

[0034] As described above, the buffer 202 can store checkpoint data. The buffer 202 can store one or more pages of the checkpoint data. The one or more pages of the checkpoint data can be used to enhance other correction methods. The buffer 202 can provide temporary storage of the data to be checkpointed from the memory dies of the stack. Details of the memory dies are illustrated and described below with respect to FIG. 3.

[0035] FIG. 3 is a block diagram of a memory die 300 of an HBM according to at least one embodiment. The memory die 300 includes multiple channel arrays 302, channel control and primary TSVs 304, and crossbar and secondary TSVs 306. The crossbar allows access to all channel arrays 302 by the secondary TSVs. The channel control blocks can receive commands through the primary TSVs of one of the channels (or one of the secondary TSVs). The channel control blocks can access pages selected by the commands and rout the data of the pages through the crossbar to one set of the TSVs shared by all. In some cases, the required bandwidth of the secondary TSVs is low, as compared to the required bandwidth of the primary TSVs. For example, if the HBM is fully checkpointed, a 16-high stack of 32GB memory dies means the transfer of 64GB in the order of 10s is 6.4GB / s. With 8Gb / s per secondary TSV, less than 10 secondary TSVs would be needed on average.

[0036] In at least one embodiment, a memory device includes a base die (e.g., base die 200) and a plurality of memory dies (e.g., multiple memory dies 300), each comprising an array of memory cells. The memory device also includes TSVs passing through theAttorney Docket No.: 27170.1090 (L1033PCT) plurality of memory devices. The base die includes a primary interface, a secondary interface, and a buffer. The base die, in response to a first command, initiates a first transfer of data from one of the plurality of memory dies into the buffer via at least one of the TSVs. The base die, in response to a second command, initiates a second transfer of data from the buffer to another device via the secondary interface or the primary interface.

[0037] In at least one embodiment, a stacked memory device includes a plurality of memory cell layers and a logic layer. The logic layer includes a first interface to couple to an external device, a second interface to couple to a fabric switch, and a buffer. The buffer is accessible by the plurality of memory layers, and the buffer is accessible by the secondary interface. The logic layer, in response to a first command, initiates a first transfer of a first block of bits from one of the plurality of memory cell layers into the buffer. The logic layer, in response to a second command, initiates a second transfer of a second block of bits from the buffer out to the fabric switch via the secondary interface.

[0038] In at least one embodiment, a memory device includes a plurality of memory dies to store data and a base die that includes a buffer for temporary storage of checkpoint data associated with the data stored in the plurality of memory dies. The base die includes a primary interface having one or more channels, a secondary interface, a control circuit and data switch. The memory device includes a plurality of TSVs coupled between the base die and the plurality of memory dies. The control circuit and data switch can receive commands through the primary interface or the secondary interface. The control circuit and data switch can route data to or from any of the primary interface, the buffer, or the secondary interface. The control circuit and data switch can route the checkpoint data to or from the buffer.

[0039] In at least one embodiment, the plurality of TSVs includes a set of primary TSVs and a set of one or more secondary TSVs. Each primary TSV is coupled between one of the one or more channels of the primary interface and one or more of the plurality of memory dies. The set of one or more secondary TSVs are coupled to the control circuit and data switch and one or more of the plurality of memory dies. In another embodiment, each primary TSV is coupled between one of the one or more channels of the primary interface and one or more of the plurality of memory dies, and the set of one or more secondary TSVs is coupled to the control circuit and data switch and all of the plurality of memory dies. In at least one embodiment, the control circuit and data switchAttorney Docket No.: 27170.1090 (L1033PCT) can route the checkpoint data to the buffer from the set of TSVs simultaneously with at least a portion of the data being transferred via the primary interface.

[0040] As described herein, the stacked memory device (or memory device or HBM) can be used for various applications that require checkpointing, such as AI / ML training, such as described in more detail below.Checkpointing in Training of Large Language Models (LLMs)

[0041] In at least one embodiment, the HBM, including the base die 200 and memory die 300, can be used in training of large language models (LLMs). Training of LLMs takes a long time, often months, and uses large clusters of compute nodes. Uncorrectable errors do occur in this situation (at least every few days, or every few hours), so checkpointing is used. Checkpointing is an active area of research since AI / ML training differs enough from HPC that the optimum approach will be different. Using the properties of AI / ML training makes effective checkpointing easier. For example, datasets for training are read from external storage and do not change during trainings. Weights are updated and in a defined state at the end of each minibatch (backpropagation complete). Activations can be recomputed starting with the forward pass of an minibatch. Parallelism (data and pipeline) means that there are copies on other nodes in the system after a node fails.

[0042] In an LLM, the number of parameters, the number of layers of the network, and the size of data used for training are all very large. Data parallelism (distributing slices of data across multiple GPUs), tensor parallelism (sharding the tensor across multiple GPUs), pipeline parallelism (distributing layers of the model across multiple GPUs), and context parallelism (breaking up long chains of tokens), are all used to meet the limits of processing by one GPU. GPUs running slices or shards are not continuously synchronized but get synchronized only at certain points of an epoch when collective operations like reduce, scatter, and gather are used to align the slices and shards. Checkpointing needs to take this into account so that a loaded checkpoint reflects a common state of all GPUs. Checkpointing should need to load only the minimum amount of information necessary to recover from an error to avoid long delays of restarting. So, weights can be checkpointed after the end of an epoch during forward pass when they do not change. The weights can be checkpointed into the buffer on a base die of the HBM. This can be done on a per-page basis with a new command from the host (either GPU or by CPU orchestrating both GPU and memory). The buffer on the base die can beAttorney Docket No.: 27170.1090 (L1033PCT) streamed out through the secondary interface. Alternatively, the per-page command could stream directly through the interfaces without buffer, but that would not allow adjusting for timing variations, e.g., when the host either has little time to interrupt for checkpoint storage, or many time slots available. An example checkpointing flow is illustrated and described below with respect to FIG. 4.

[0043] FIG. 4 is a flow diagram of a checkpointing process flow 400 according to at least one embodiment. The checkpointing process flow 400 can be part of an AI / ML training. The checkpointing process flow 400 can be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions run on a processing device to perform hardware simulation), or a combination thereof. In one embodiment, the checkpointing process flow 400 is performed by a processing unit, such as a CPU, a GPU, a DPU, or the like, in connection with an HBM having a primary interface, a secondary interface, and a buffer as described above with respect to FIG. 1 to FIG. 3.

[0044] Referring to FIG. 4, the checkpointing process flow 400 begins with the processing logic starting an epoch (block 402). At block 402, the processing logic resets a checkpointing counter, sets a write buffer of the HBM to A and a read buffer of the HBM to B. As described above, the buffer 202 can be a double buffer with two portions that are alternately written to from the memory stack and read to the secondary interface. At block 406, the processing logic begins a forward pass of the AI / ML training. At block 408, the processing logic issues a checkpoint command (e.g., first command) to the HBM to save all weights during the forward pass. Checkpointing saves all weights during a forward pass since weights are unchanged in the forward pass. This allows recovery to start at the beginning of a forward pass.

[0045] At block 410, the HBM receives the checkpoint command. The checkpoint write command can be received over the primary interface or the secondary interface. In at least one embodiment, a GPU memory controller can issue checkpoint write commands to each channel over the primary interface. At block 412, the HBM stores addressed page in the write buffer. At block 414, the HBM determine if the write buffer is full. If not full, the HBM returns to block 410. Once the write buffer is full at block 414, the role of the write buffer and the read buffer are swapped (block 416). Also, once the write buffer is full at block 414, the HBM transfers the read buffer through the secondary interface to the processing unit. The checkpoint read commands can be received over the fabric.Attorney Docket No.: 27170.1090 (L1033PCT)

[0046] At block 420, the processing logic determines if the forward pass has ended. If the forward pass has not ended, the processing logic returns to block 408. If the forward pass has ended at block 420, the processing logic performs the back propagation and optimizer of the AI / ML training (block 422). At block 424, the processing logic determines if the epoch has ended. If not, the processing logic returns to block 422. If the epoch has ended at block 424, the processing logic ends the checkpointing process flow 400.

[0047] While the example given here is for AI / ML training, the HBM can also be used for checkpointing in HPC other processes. The timings and transfer sizes can be adjusted according to the application.

[0048] FIG. 5 is a block diagram of a stacked memory device 500 with a base layer 502 and multiple memory layers 504 according to at least one embodiment. The base layer 502 includes a primary interface 506 and a secondary interface 508. The primary interface 506 is coupled to multiple primary TSVs 514. The secondary interface 508 is coupled to a secondary TSV 516. Each of the primary TSVs 514 is coupled to a memory layer 504 in each of the two ranks illustrated in FIG. 5. As illustrated, the stacked memory device 500 includes a first rank 510 and a second rank 512. Alternatively, the stacked memory device 500 can be organized as one rank or more than two ranks. The secondary TSV 516 is coupled to all memory layers 504 since the bandwidth requirement is lower than the primary interface 506. As such, the secondary TSV 516 connects to all memory layers 504 in the stack, while the primary TSVs 514 connect to only one layer in each rank. The secondary interface 508 can connect to an external interface, such as a fabric interface that follows the network standard of system, such as NVLink, InfiniBand, or Ethernet. In that way, the fabric interface can operate independently from the GPU (or other processing unit coupled to the stacked memory device 500). While the secondary interface 508 could also be used to read a stored checkpoint, the secondary interface 508 will typically have less bandwidth than the primary interface 506. The GPU (or other processing unit) will not be able to work after an error until the checkpoint is restored. So, a better approach may be to use the primary interface 506 for restoration of the checkpoint due to its higher bandwidth. The secondary interface 508 can be used to restore one or more pages from a checkpoint to enhance other error correction methods performed by a connected processing unit (e.g., GPU, CPU, etc ).Attorney Docket No.: 27170.1090 (L1033PCT)

[0049] In some embodiments, the secondary TSV 516 can be avoided if the primary TSVs 514 are used to transfer data from the memory base layer 502 to a data switch on the base layer 502. This would, however, block one channel for the duration of a page transfer every time a checkpoint command is issued to that channel.

[0050] In some embodiments, additional feedback from the base die of the HBM can be provided through either the primary interface or the secondary interface. The additional feedback can create backpressure if buffer overflow is expected to occur. For example, there can be an in-stack counter selecting the page to send to the buffer so that a memory controller only needs to issue the command, not the target page. The base die can receive a counter-reset command to reset the in-stack counter. In other embodiments, the base die can have a skip table so that pages that do not need to be checkpointed are skipped. For example, the pages that do not contain weights would be skipped for checkpointing.

[0051] In some embodiments, the fabric interface can be used to restore data from one HBM to a different HBM. For example, the checkpoint data can be stored in CPU DRAM (e.g., DDR DRAM 118), fabric DRAM (e.g., fabric attached DRAM 114), or storage 116. The CPU 104 can move the checkpoint data between these locations and sort them to expedite recovery. In some embodiments, restoring over the primary interface is most likely, as the primary interface is likely higher bandwidth. This can be limited by the speed of where the data is being restored from and the link to that location. An example of the checkpoint data is illustrated and described below with respect to FIG. 6.

[0052] FIG. 6 illustrates an example format of checkpoint data 600 according to at least one embodiment. The checkpoint data 600 can have a header 602 uniquely identifying a location in the stack, and parity data 606. The parity data 606 can be the original parity of the memory die or newly calculated parity data to cover the header 602 as well. In at least one embodiment, the header 602 includes 32 bits, including 2 bits for a stack identifier (rank), 2 bits for a die, 4 bits for a pseudo channel (PC), 4 bits for a bank address (BA), 14b for a row address (RA), and 6 bits reserved for future use (RFU). In at least one embodiment, the one atom data 604 includes lkB-4kB of data, representing one or more pages of data. The parity data 606 can be 128B, 512B, or the like.

[0053] In some embodiments, the checkpoint commands to perform checkpointing can be issued less often than refresh commands of the HBM. For example, a transfer of a full HBM by one or more pages simultaneously is possible within a forward pass (e.g., IT parameter model with 3,072 GPUs: forward pass 15 seconds (s), backpropagation 30s).Attorney Docket No.: 27170.1090 (L1033PCT)A transfer of a page can take 210 nanoseconds (ns) (e.g., 5 column addresses, tCCDL=5ns, tRC=50ns). Using a 32Gb HBM with 4 die per rank and 4 ranks, and 16 PC per die, 4 bank addresses, 14 row addresses per PC, would be 14.1s transfer time page after page (i.e., 22• 22• 24• 24• 214= 6.71 • 107pages = 14.1s transfer time page after page). Transfer of full HBM by one or a few pages simultaneously is possible within forward pass 15s. In this example, the SRAM size, in 7nm technology, is 6mm2for 64Mb. There can be two 32Mb buffers (one is written to from the stack while the other is read through the secondary interface. There can be 32 memory controllers (one for each channel), and each memory controller can issue, on average, a page transfer command every 6.72 microseconds (i.e., 210ns * 32 = 6.72ps). This is about half as often as the need to issue refresh commands. Each memory controller can have 1Mb in each buffer (i.e., 128Mb or 16k pages per rank and PC). The buffers allow flexibility in aligning checkpoint data from the memory die to the secondary interface.

[0054] In some cases, the checkpoint commands can be “hidden behind" the refresh commands. Additional details of hiding the checkpoint commands behind the refresh commands are described below with respect to FIG. 7. Checkpointing commands can be hidden behind other commands, such as calibration operations.

[0055] FIG. 7 is a flow diagram of a method 700 of checkpointing in connection with refresh operations according to at least one embodiment. The method 700 can be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions run on a processing device to perform hardware simulation), or a combination thereof. In one embodiment, the method 700 is performed by a processing unit, such as a CPU, a GPU, a DPU, or the like, in connection with an HBM having a primary interface, a secondary interface, and a buffer as described above with respect to FIG. 1 to FIG. 3.

[0056] It should be noted that refresh operations can affect multiple rows M (e.g., 4 or 8 rows) in a bank. When checkpointing behind a refresh operation, the checkpoint bytes from one of the AT rows can be affected by the refresh operation, so the frequency of the checkpoint operations can be set per a refresh frequency, F, stored in a configuration register. A counter can count cycles from 0 to M-l, where the counter indicates which of the AT rows is to be checkpointed. The selected row can be refreshed first in the group of AT rows to be refreshed. This allows the full refresh cycle time (tRFC window) for the selected row to have its data transferred to the buffer on the base die. Checkpoint operations can operate on a complete page (e.g., DRAM page), or a subset of a page. AnAttorney Docket No.: 27170.1090 (L1033PCT)Internal Row Counter IRC) can be used to cycle through all rows in a bank, checkpointing them at a configurable frequency. It should be noted that the memory dies can include additional counters per bank that help manage the frequency of the checkpointing.

[0057] Referring to FIG. 7, the method 700 begins with the processing logic initialing a frequency counter, / , and a row counter, m, in a group of AY rows refreshed by a refresh operation (block 702). That is, the counters / and m are set equal to zero. The processing logic wait for the next refresh command (block 704). If the processing logic determines that the refresh command is received at block 706, the processing logic increments the frequency counter, f (block 708). At block 710, the processing logic determines if the frequency counter, is equal to a refresh frequency, F (i.e., / =F?). If so, the processing logic returns to block 704. If not, the processing logic sets the counter / to zero (block 712) and the logic checkpoints bytes from row m in the group of AT rows being refreshed (block 714). The address of this row is a function of bank number and IRC. The memory die can choose any M rows in the bank to be refreshed by each refresh operation. A variable for the IRC can be set to the row address to be checkpointed (one of the AT rows refreshed). The IRC can be a function of the row address being checkpointed, the bank group, bank address, and the channel address. An example physical address for a checkpointed row (i.e., IRC) is shown below for a 32GB DRAM with 16 channels, 4 Bank Groups, 8 Banks per Bank Group, 64K Rows per Bank, and 1KB DRAM pages.

[0058] After block 714, the processing logic increments the counter m (block 716). The processing logic determines if the counter m is equal to AT (i.e., m=M?). If not, the processing logic returns to block 704. If the counter m is equal to AT at block 718, the processing logic resets the counter m to zero, before returning to block 704. In this manner, the processing logic can rotate through all AT rows, using a counter m that indicates which of the AT rows should be refreshed first. An example refresh operation that refreshes 4 rows per bank is illustrated and described below with respect to FIG. 8.

[0059] FIG. 8 illustrates an example refresh operation that refreshes 4 rows per bank according to at least one embodiment. As described above, a refresh operation affects multiple rows (e.g., M) per bank. In this example, there are 4 rows per bank. WithAttorney Docket No.: 27170.1090 (L1033PCT) checkpointing being hidden behind a refresh operation, the checkpointed row can be refreshed first so it has the maximum time (e.g., tRFC) to be written to the checkpoint buffer. The counter m can be used to rotate through all Arrows, where the counter m, indicates which of the Arrows should be refreshed first. For example, when the counter m, is equal to zero, a first row (Row 0) is the checkpointed row, and when the counter m, is equal to one, a fifth row (Row 4) is the checkpointed row.

[0060] FIG. 9 is a flow diagram of a method 900 of checkpointing in connection with executing a program element according to at least one embodiment. The method 900 can be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions run on a processing device to perform hardware simulation), or a combination thereof. In one embodiment, the method 900 is performed by a processing unit, such as a CPU, a GPU, a DPU, or the like, in connection with an HBM having a primary interface, a secondary interface, and a buffer as described above with respect to FIG. 1 to FIG. 3.

[0061] Referring to FIG. 9, the method 900 begins with the processing logic executing a program element (block 902). The processing logic can save a checkpoint (block 904). The processing logic determines if the program element is done executing at block 906. If the program element is not done executing, the processing logic continues with saving checkpoints periodically as described herein. While executing the program element, the processing logic also performs ongoing error detection (block 908). If an error is detected at block 910, the processing logic determines whether the correctable (block 912). If the error is correctable, the processing logic corrects the error and returns to performing the ongoing error detection at block 908. Similarly, if no error is detected at block 910, the processing logic returns to the ongoing error detection at block 908. If the error is not correctable at block 912, the processing logic loads the checkpoint saved at block 904 (block 914), and returns to continue with the program element with the restored checkpoint being loaded.

[0062] FIG. 10 is a flow diagram of a method 1000 of operating a stacked memory device with primary and secondary interfaces according to at least one embodiment. The method 1000 may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions run on a processing device to perform hardware simulation), or a combination thereof. In one embodiment, the method 1000 is performed by any of the hardware described above with respect to FIG. 1 to FIG. 9. In at least one embodiment,Attorney Docket No.: 27170.1090 (L1033PCT) the method 1000 is performed by an HBM 108 of FIG. 1. In at least one embodiment, the method 1000 is performed by a base die 200 of FIG. 2. In at least one embodiment, the method 1000 is performed by the stacked memory device 500 of FIG. 5.

[0063] Referring to FIG. 10, the method 1000 begins with the processing logic sending or receiving data from or to the plurality of memory dies over a primary interface of the base die (block 1002). At block 1004, the processing logic receives a first command through the primary interface or a secondary interface of the base die. At block 1006, the processing logic initiates, in response to the first command, a first transfer of a first block of bits from one of the plurality of memory dies to a buffer on the base die. At block 1008, the processing logic receives a second command through the primary interface or the secondary interface. At block 1010, the processing logic initiates, in response to the second command, a second transfer of a second block of bits from the buffer out of the stacked memory device via the secondary interface or the primary interface. The second transfer occurs concurrently with the sending or receiving the data.

[0064] In a further embodiment, the processing logic receives a third command through the primary interface or the secondary interface. The processing logic initiates, in response to the third command, a third transfer of checkpoint data stored in the buffer over the secondary interface.

[0065] In a further embodiment, the processing logic receives a third command through the primary interface or the secondary interface. The processing logic initiates, in response to the third command, a third transfer of checkpoint data stored in the buffer over the primary interface.

[0066] In at least one embodiment, the buffer includes a first portion and a second portion, and the processing logic writes first data to the first portion at a first time and reads second data from the second portion at the first time. The processing logic writes third data to the second portion at a second time and reads fourth data form the first portion at the second time.

[0067] In a further embodiment, the processing logic outputs one or more pages of checkpoint data from the buffer to an error correction method executing on a processing unit coupled to the stacked memory device.

[0068] FIG. 11 illustrates an expansion card 1100 with eight GPUs and multiple HBMs coupled to the eight GPUs according to at least one embodiment. The expansion card 1100 can include eight GPUs and six HBMs per GPU. The HBMs can be coupled to the GPU using a primary interface and a fabric interface using the secondary interface asAttorney Docket No.: 27170.1090 (L1033PCT) described herein. The expansion card 1100 can be used in a computer, such as coupled to a motherboard having a central processing unit (CPU). The expansion card 1100 can have other numbers of GPUs, and different configurations, such as described herein. The expansion card 1100 can be used with other expansion cards in a server. Multiple servers can be used in a rack, such as illustrated in FIG. 12.

[0069] FIG. 12 illustrates a rack 1200 with multiple servers with multiple expansion cards according to at least one embodiment. The rack 1200 can include multiple servers (four illustrated), each server including multiple expansion cards, such as described above with respect to FIG. 11. For example, a first server 1202 can include eight expansion cards coupled to a server board (e.g., server motherboard). The rack 1200 can include 256 GPUs, where there are four servers, each server having eight expansion cards, each expansion card having eight GPUs. Alternatively, the rack 1200 can have other numbers of servers and other numbers of expansion cards with different configurations of GPUs.

[0070] It is to be understood that the above description is intended to be illustrative and not restrictive. Many other implementations will be apparent to those of skill in the art upon reading and understanding the above description. Therefore, the disclosure scope should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

[0071] In the above description, numerous details are set forth. It will be apparent, however, to one skilled in the art that the aspects of the present disclosure may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the present disclosure.

[0072] Some portions of the detailed descriptions above are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self- consistent sequence of steps leading to the desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenientAttorney Docket No.: 27170.1090 (L1033PCT) at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0073] However, it should be borne in mind that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “receiving,” “determining,” “selecting,” “storing,” “setting,” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system’s registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0074] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable ROMs (EPROMs), electrically erasable programmable ROMs (EEPROMs), magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

[0075] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatuses to perform the required method steps. The required structure for a variety of these systems will appear as set forth in the description. In addition, aspects of the present disclosure are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present disclosure as described herein.

[0076] Aspects of the present disclosure may be provided as a computer program product, or software, that may include a machine-readable medium having stored thereon instructions, which may be used to program a computer system (or other electronicAttorney Docket No.: 27170.1090 (L1033PCT) devices) to perform a process according to the present disclosure. A machine-readable medium includes any procedure for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium (e.g., read-only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory devices, etc.).

Claims

Attorney Docket No.: 27170.1090 (L1033PCT)CLAIMSWhat is claimed is:

1. A stacked memory device comprising: a base layer having a primary interface, a secondary interface, and a buffer; and a plurality of memory layers, wherein the buffer is accessible by the plurality of memory layers, wherein the buffer is accessible by the secondary interface, wherein: the base layer, in response to a first command, initiates a first transfer of a first block of bits from a memory layer of the plurality of memory layers into the buffer; the base layer, in response to a second command, initiates a second transfer of a second block of bits from the buffer out of the stacked memory device via the secondary interface; and the second transfer occurs concurrently with a third transfer of data over the primary interface.

2. The stacked memory device of claim 1, wherein the buffer comprises a first portion and a second portion, wherein the first portion and the second portion alternate between being written to and being read from.

3. The stacked memory device of claim 1, wherein the first command is received over the primary interface, and wherein the second command is received over the secondary interface.

4. The stacked memory device of claim 1, wherein the secondary interface is coupled to a fabric interface of a computing system in which the stacked memory device resides, and wherein the fabric interface operates according to a network standard of the computing system.

5. The stacked memory device of claim 1, wherein the secondary interface is coupled to a fabric interface of a computing system in which the stacked memory device resides, and wherein the fabric interface operates independently from a processing unit of the computing system.Attorney Docket No.: 27170.1090 (L1033PCT)6. The stacked memory device of claim 1, wherein the primary interface comprises a first bandwidth, wherein the secondary interface comprises a second bandwidth less than the first bandwidth.

7. The stacked memory device of claim 6, wherein, in response to a third command to read a stored checkpoint stored in the buffer, the stored checkpoint is read via the secondary interface.

8. The stacked memory device of claim 6, wherein, in response to a third command to read a stored checkpoint stored in the buffer, the stored checkpoint is read via the primary interface.

9. The stacked memory device of claim 1, wherein the buffer to store one or more pages of a checkpoint data.

10. The stacked memory device of claim 1, wherein the base layer further comprises: channel logic coupled to a number of channels of the primary interface and a set of primary Through-Silicon Vias (TSVs) coupled between the base layer and the plurality of memory layers; a set of one or more secondary TSVs coupled between the base layer and the plurality of memory layers; and secondary interface control logic and data switch coupled to the channel logic, the set of primary TSVs, the set of one or more secondary TSVs, the buffer, and the secondary interface, wherein the base layer, in response to the first command, initiates the first transfer of the first block of bits from one of the plurality of memory layers into the buffer through the set of one or more secondary TSVs and the data switch.

11. The stacked memory device of claim 1, wherein the base layer further comprises: channel logic coupled to a number of channels of the primary interface and a set of primary Through-Silicon Vias (TSVs) coupled between the base layer and the plurality of memory layers; and secondary interface control logic and data switch coupled to the channel logic, the set of primary TSVs, the buffer, and the secondary interface, wherein the base layer, in response to the first command, initiates the first transfer of the first block of bits fromAttorney Docket No.: 27170.1090 (L1033PCT) one of the plurality of memory layers into the buffer through the set of primary TSVs and the data switch.

12. A memory device comprising: a plurality of memory dies to store data; a base die comprising: a buffer for temporary storage of checkpoint data associated with the data stored in the plurality of memory dies; a primary interface having one or more channels; a secondary interface; a control circuit and data switch; and a plurality of Through- Silicon Vias (TSVs) coupled between the base die and the plurality of memory dies, wherein the control circuit and data switch is to: receive commands through the primary interface or the secondary interface; route data to or from any of the primary interface, the buffer, or the secondary interface; and route the checkpoint data to or from the buffer.

13. The memory device of claim 12, wherein the plurality of TSVs comprises: a set of primary TSVs, each primary TSV being coupled between one of the one or more channels of the primary interface and one or more of the plurality of memory dies; and a set of one or more secondary TSVs coupled to the control circuit and data switch and one or more of the plurality of memory dies.

14. The memory device of claim 12, wherein the plurality of TSVs comprises: a set of primary TSVs, each primary TSV being coupled between one of the one or more channels of the primary interface and one or more of the plurality of memory dies; and a set of one or more secondary TSVs coupled to the control circuit and data switch and all of the plurality of memory dies.Attorney Docket No.: 27170.1090 (L1033PCT)15. The memory device of claim 12, wherein the control circuit and data switch is to route the checkpoint data to the buffer from the plurality of TSVs simultaneously with at least a portion of the data being transferred via the primary interface.

16. A method of operating a stacked memory device comprising a base die and a plurality of memory dies, the method comprising: sending or receiving data from or to the plurality of memory dies over a primary interface of the base die; receiving a first command through the primary interface or a secondary interface of the base die; initiating, in response to the first command, a first transfer of a first block of bits from one of the plurality of memory dies to a buffer on the base die; receiving a second command through the primary interface or the secondary interface; and initiating, in response to the second command, a second transfer of a second block of bits from the buffer out of the stacked memory device via the secondary interface or the primary interface, wherein the second transfer occurs concurrently with the sending or receiving the data.

17. The method of claim 16, further comprising: receiving a third command through the primary interface or the secondary interface; and initiating, in response to the third command, a third transfer of checkpoint data stored in the buffer over the secondary interface.

18. The method of claim 16, further comprising: receiving a third command through the primary interface or the secondary interface; and initiating, in response to the third command, a third transfer of checkpoint data stored in the buffer over the primary interface.

19. The method of claim 16, wherein the buffer comprises a first portion and a second portion, wherein the method comprises: writing first data to the first portion at a first time;Attorney Docket No.: 27170.1090 (L1033PCT) reading second data from the second portion at the first time; writing third data to the second portion at a second time; and reading fourth data form the first portion at the second time.

20. The method of claim 16, further comprising outputting one or more pages of checkpoint data from the buffer to an error correction method executing on a processing unit coupled to the stacked memory device.

Images