Page exit techniques for multi-page DRAM failure

Abstract

A processing system (100) employs techniques for enhancing dynamic random access memory (DRAM) page exit to facilitate identification and exit of pages affected by a multi-page DRAM failure. In response to detecting an uncorrectable error at a first page of the DRAM, the processing system identifies a second page of the DRAM for potential exit based on one or more of: physical proximity to the first page; including in an address range stored at a fault map (510) that tracks addresses of the DRAM page having the detected fault; and predicting a set of pages to check for a fault based on a miss at a translation lookaside buffer (TLB) (120).

Description

BACKGROUND OF THE INVENTION

[0001] Computer memories such as dynamic random access memories (DRAMs) are vulnerable to various transient and permanent faults. Transient faults result in incorrect data being read from a memory location until that location is overwritten with correct data and do not indicate device damage, while permanent faults (i.e., hard faults) cause the memory location to continuously return an incorrect value. In response to encountering a permanent fault (referred to as a detectable uncorrectable error (DUE)), the processing system takes the affected portion of the memory out of service. SUMMARY OF THE INVENTION

[0002] In the examples described herein, techniques are provided for enhancing DRAM page retirement in a processing system to facilitate identification and retirement of pages affected by multi-page DRAM faults. In one example implementation, a method includes identifying a second page of a DRAM for potential retirement in response to detecting an uncorrectable error at a first page of the dynamic random access memory (DRAM).

[0003] In some implementations, identifying the second page includes identifying a subset of pages of the DRAM at a first physical distance from the first page, the second page being included in the subset of pages. Identifying also includes checking for the presence of faults at the subset of pages. In some implementations, the method further includes retiring the second page in response to detecting an uncorrectable error at the second page. Detecting the uncorrectable error at the first page can be performed while checking the DRAM for faults.

[0004] In some implementations, the method further includes recording the uncorrectable error detected at the first page at a fault map including a plurality of page addresses of pages having faults. Identifying the second page can include scanning the fault map for page addresses adjacent to the first page in response to recording the uncorrectable error at the first page at the fault map. In some implementations, the method further includes selecting a subset of page addresses for retirement based on the presence of uncorrectable errors at one or more pages in the subset, the subset of page addresses including the page address of the second page. Identifying the second page can include: identifying a subset of pages of the DRAM predicted to be accessed based on a miss at a translation lookaside buffer (TLB), the second page being included in the subset of pages; and retiring the subset of pages in response to detecting an uncorrectable error at the second page.

[0005] In another example, a method includes exiting a first page of a dynamic random access memory (DRAM) before accessing the first page in response to detecting an uncorrectable error at a second page of the DRAM. In some implementations, the method further includes: identifying a subset of pages of the DRAM that are at a first physical distance from the second page, the first page being included in the subset of pages; and checking for the presence of a fault at the subset of pages. The method may also include exiting the first page in response to detecting an uncorrectable error at the second page.

[0006] In some implementations, the method further includes detecting an uncorrectable error at the first page while checking the DRAM for faults. The method may also include recording the uncorrectable error detected at the first page at a fault map that includes a plurality of page addresses of pages having faults. In some implementations, the method further includes scanning the fault map for page addresses adjacent to the second page in response to recording the uncorrectable error at the second page at the fault map.

[0007] The method may include selecting a subset of page addresses to exit based on the presence of an uncorrectable error at one or more pages in the subset, the subset of page addresses including the page address of the first page. In some implementations, the method further includes: identifying a subset of pages of the DRAM to be accessed that are predicted based on a miss at a translation lookaside buffer (TLB), the first page being included in the subset of pages; and exiting the subset of pages in response to detecting an uncorrectable error at the first page.

[0008] In another example, a device includes a dynamic random access memory (DRAM) and control logic configured to identify a second page of the DRAM for potential exit in response to detecting an uncorrectable error at a first page of the DRAM. In some implementations, the control logic is further configured to identify a subset of pages of the DRAM that are at a first physical distance from the first page, the second page being included in the subset of pages, and to check for the presence of a fault at the subset of pages.

[0009] The control logic circuit may be further configured to record an uncorrectable error detected at a first page at a fault map including a plurality of page addresses of pages having faults, and to scan the fault map for page addresses adjacent to a second page in response to recording an uncorrectable error at the second page at the fault map. The control logic circuit may also be configured to select a subset of page addresses for eviction based on the presence of uncorrectable errors at one or more pages in the subset, the subset of page addresses including the page address of the first page. In some embodiments, the control logic circuit is further configured to identify a subset of pages of the DRAM to be accessed predicted based on a miss at a translation lookaside buffer (TLB), the second page being included in the subset of pages, and to evict the subset of pages in response to detecting an uncorrectable error at the second page. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present disclosure may be better understood by reference to the accompanying drawings, and its numerous features and advantages will be apparent to those skilled in the art. Like reference numerals are used in the different drawings to denote similar or identical items.

[0011] Figure 1 is a block diagram of a processing system according to some embodiments, the processing system being configured to implement a page eviction policy to proactively identify pages of a memory for eviction based on detected faults in other pages of the memory.

[0012] Figure 2 is a block diagram of a control logic circuit according to some embodiments, the control logic circuit being configured to implement a page eviction policy in conjunction with a patrol scrubber at a system memory of a processing system.

[0013] Figure 3 is a block diagram of a page eviction policy according to some embodiments for proactively identifying pages of a memory for eviction based on detected faults in other pages of the memory.

[0014] Figure 4 is a flowchart illustrating a method of identifying faults at a subset of pages of a DRAM based on detecting an uncorrectable error at a first page of the DRAM according to some embodiments.

[0015] Figure 5 is a block diagram of a patrol scrubber for identifying faults at a subset of pages of a DRAM based on a fault map storing addresses of pages having detected uncorrectable errors according to some embodiments.

[0016] Figure 6 is a flowchart illustrating a method of identifying faults at a subset of pages of a DRAM based on a fault map storing addresses of pages having detected uncorrectable errors according to some embodiments.

[0017] Figure 7 is a block diagram of control logic circuitry that identifies a subset of pages of a DRAM for erasure based on misses at a translation lookaside buffer according to some embodiments.

[0018] Figure 8 is a diagram illustrating virtual address to physical address translation for identifying a subset of pages of a DRAM for erasure based on misses at a translation lookaside buffer according to some embodiments.

[0019] Figure 9 is a diagram illustrating extraction of a set index from a virtual page offset for identifying a subset of pages of a DRAM for erasure based on misses at a translation lookaside buffer according to some embodiments.

[0020] Figure 10 is a diagram illustrating an example of a set index according to some embodiments, where the number of page offset bits is less than the total number of bits required for the set index and offset for identifying a subset of pages of a DRAM for erasure based on misses at a translation lookaside buffer.

[0021] Figure 11 is a flowchart of a method for identifying a subset of pages of a DRAM for erasure based on misses at a translation lookaside buffer according to some embodiments. DETAILED DESCRIPTION

[0022] Conventional processing systems typically employ one or more memory modules having an array of memory cells (e.g., industry standard memories such as dual in-line memory modules (DIMMs)). A DIMM consists of rows of DRAM chips, which in turn are organized into banks of rows and columns of memory cells configured to store data. Conventional eviction algorithms evict portions of memory affected by DUE at the granularity level of a single page (i.e., 4 kilobytes), which typically maps to a single DRAM row. A page is a fixed-length contiguous block of virtual memory, described by a single entry in a page table. A memory page is the smallest data unit for memory management in a virtual memory operating system.

[0023] However, larger granularity faults in DRAM can affect multiple rows or an entire DRAM bank. Based on how data is interleaved in the memory system, a single page eviction action (whether in response to DUE or initiated in a corrected error pattern) typically evicts a single row or a portion of a DRAM row. Thus, a DUE spanning multiple pages of memory (such as a multi-bit fault in high bandwidth memory (HBM2 / 2E) or an unbounded fault in HBM3) may require multiple calls to the drive of the processing system to initiate page eviction, negatively impacting system availability and performance.

[0024] Processing systems typically use virtual memory to handle data access by executing programs (e.g., applications, operating systems, device drivers, etc.). In such a processing system, a program uses "virtual addresses" in a "virtual address space" to access memory, rather than using an address based on the physical location (or "physical address") of a memory-based page. These "virtual address spaces" are local address spaces dedicated to the corresponding program. Thus, to support memory access, a processing system typically employs an address translation circuit to translate virtual addresses into corresponding physical addresses. The address translation circuit uses one or more translation lookaside buffers (TLBs) to cache virtual-to-physical address translations for efficient lookup by the processor core.

[0025] Figures 1 to 11 Techniques are illustrated for enhancing DRAM page eviction in a processing system to facilitate identification and eviction of pages affected by multi-page DRAM failures. In response to detecting an uncorrectable error at a first page of a DRAM, the processing system identifies one or more additional pages (also collectively referred to herein as "second pages") of the DRAM for potential eviction based on one or more of the following: physical proximity to the first page; inclusion within an address range stored at a fault map tracking addresses of DRAM pages with detected failures; and prediction of a set of pages to check for faults based on a miss at the TLB. By identifying additional pages of the DRAM for potential eviction based on the DUE at the first page, the processing system proactively evicts pages affected by multi-page failures and reduces calls by the driver to initiate page eviction.

[0026] In some embodiments, the processing system employs a page eviction policy that specifies that in response to encountering a DUE at a first page of a DRAM, the processing system erases a subset of pages within a pre-determined physical distance from the first page for the presence of a large (multi-page) failure (i.e., checks for errors at the subset of pages). The pre-determined physical distance is set based on the page eviction policy. If the processing system encounters a failure at the subset of pages, the processing system evicts the subset of pages.

[0027] In other embodiments, the processing system employs a page eviction policy that specifies that in response to encountering a DUE at a first page of a DRAM, the processing system adds the address of the page with the DUE to a fault map maintained by a sweep eraser. The sweep eraser scans the DRAM at regular intervals to address faults and lists the addresses of page faults at the fault map. The processing system determines whether the address of the first page falls within a set of pages identified as falling within a large fault in the fault map. If the address of the first page is within the address range of the large fault, the processing system evicts all pages within that range.

[0028] In some embodiments, the processing system employs a page eviction strategy that identifies pages for erasure based on misses at the TLB. The TLB miss predicts a miss at the cache that will require a memory access. If the sweep eraser encounters a DUE at a page included in a subset of adjacent pages (referred to herein as the "miss set") predicted to miss at the cache, the processing system evicts the entire subset of adjacent pages. In some embodiments, the processing system identifies the miss set as pages having the same set index and offset bit values. In response to a miss at the TLB, the processing system uses a hash function to decode the page offset of the virtual address of the TLB miss and returns the index of the set. The processing system inserts the set number into a table to track the set referenced by the TLB miss. When the number of misses for a set exceeds a threshold, the sweep eraser erases the pages in the set (i.e., reads and checks for errors within those pages) to identify any DUEs within the set.

[0029] Figure 1 is a block diagram of a processing system according to some embodiments, the processing system being configured to implement a page eviction strategy to proactively identify pages of memory for eviction based on detected faults in other pages of the memory. Processing system 100 includes a central processing unit (CPU) 102 and a parallel processing unit (PPU) 104, also referred to herein as parallel processor 104. In various embodiments, CPU 102 includes one or more single-core or multi-core CPUs. In various embodiments, parallel processor 104 includes any collaborative collection of hardware and / or software that, relative to resources such as a conventional CPU, a conventional graphics processing unit (GPU), and combinations thereof, performs functions and computations associated with accelerated graphics processing tasks, data parallel tasks, nested data parallel tasks in an accelerated manner. In Figure 1 embodiments, processing system 100 is formed on a single silicon die or package that combines CPU 102 and parallel processor 104 to provide a unified programming and execution environment. This environment enables parallel processor 104 to be used as smoothly as CPU 102 for some programming tasks. In other embodiments, CPU 102 and parallel processor 104 are separately formed and mounted on the same or different substrates. It should be understood that processing system 100 may include one or more software, hardware, and firmware components in addition to or different from those Figure 1 shown in Figure 1 For example, processing system 100 may additionally include one or more input interfaces, non-volatile storage devices, one or more output interfaces, network interfaces, and one or more displays or display interfaces. Processing system 100 includes, for example, servers, desktop computers, laptop computers, tablet computers, mobile phones, gaming consoles, and the like.

[0030] As Figure 1 shown, the processing system 100 also includes a system memory 106, an operating system 108, a communication infrastructure 130, and one or more applications 114. Access to the system memory 106 is managed by a memory controller (not shown) coupled to the system memory 106. For example, requests from the CPU 102 or other devices to read from or write to the system memory 106 are managed by the memory controller. In some embodiments, one or more of the applications 114 include various programs or commands for performing computations that are also performed at the CPU 102. The CPU 102 sends selected commands for processing at the parallel processor 104. The operating system 108 and the communication infrastructure 130 will be discussed in more detail below. The processing system 100 also includes device drivers 116 and a memory management unit, such as an input / output memory management unit (IOMMU) 118. The components of the processing system 100 may be implemented as hardware, firmware, software, or any combination thereof. In some embodiments, the processing system 100 includes one or more software, hardware, and firmware components in addition to or different from those shown in Figure 1 those shown in Figure 1 and different from those shown in

[0031] Within the processing system 100, the system memory 106 includes non-persistent memory, such as DRAM (not shown). In various embodiments, the system memory 106 stores processing logic instructions, constant values, variable values during the execution of portions of applications or other processing logic, or other required information. For example, in various embodiments, portions of the control logic for performing one or more operations on the CPU 102 reside in the system memory 106 during the execution of the corresponding portions of the operation by the CPU 102. During execution, the corresponding applications, operating system functions, processing logic commands, and system software reside in the system memory 106. The underlying control logic commands of the operating system 108 typically reside in the system memory 106 during execution. In some embodiments, other software commands (e.g., device drivers 116) also reside in the system memory 106 during the execution of the processing system 100.

[0032] The system memory 106 includes a page table 128 that maintains a record of page table entries that store virtual address to physical address translation information for pages of data stored in the system memory. When a request to access memory at a given virtual address is received from a program, the CPU 102 or the parallel processor 104 performs a page table walk to obtain the corresponding physical address information of the page table entry from the page table 128 for providing the physical address associated with the virtual address.

[0033] The IOMMU 118 is a multi-context memory management unit. As used herein, a context is considered the environment within which the kernel executes and the domain within which synchronization and memory management are defined. A context includes: a set of devices; the memory accessible to those devices; the corresponding memory attributes; and one or more command queues for scheduling the execution of the kernel or the execution of operations on memory objects. The IOMMU 118 includes logic for performing virtual-to-physical address translation for memory page accesses of a device such as the parallel processor 104. In some embodiments, the IOMMU 118 also includes or has access to a translation lookaside buffer (TLB) 120. As an example, the TLB 120 is implemented in content addressable memory (CAM) to accelerate the translation of logical (i.e., virtual) memory addresses to physical memory addresses for requests made by the parallel processor 104 to data in the system memory 106. The TLB 120 stores a subset of virtual addresses to physical address information stored at the page table 128. In some embodiments, the TLB 120 is implemented as a hierarchy of multiple TLBs.

[0034] In various embodiments, the communication infrastructure 130 interconnects the components of the processing system 100. The communication infrastructure 130 includes (not shown) one or more of the following: a Peripheral Component Interconnect (PCI) bus, an Extended PCI (PCI-E) bus, an Advanced Microcontroller Bus Architecture (AMBA) bus, an Advanced Graphics Port (AGP), or other such communication infrastructure and interconnects. In some embodiments, the communication infrastructure 130 also includes an Ethernet network or any other suitable physical communication infrastructure that meets the data transfer rate requirements of the application. The communication infrastructure 130 also includes functionality for interconnecting components, including the components of the processing system 100.

[0035] A driver, such as device driver 116, communicates with a device (e.g., parallel processor 104) via an interconnect or communication infrastructure 130. When a program calls a routine in device driver 116, device driver 116 issues a command to the device. Once the device sends data back to device driver 116, device driver 116 calls the routine in the original calling program. Generally, a device driver is a hardware - specific and operating - system - specific driver to provide interrupt handling required for any necessary asynchronous time - related hardware interfaces. In some embodiments, compiler 126 is embedded within device driver 116. Compiler 126 compiles source code into program instructions required for execution by processing system 100. During such compilation, compiler 126 applies transformations to the program instructions at various compilation stages. In other embodiments, compiler 126 is a stand - alone application. In various embodiments, device driver 116 controls the operation of parallel processor 104 by, for example, providing an application programming interface (API) to software (e.g., application 114) executing at CPU 102 to access various functions of parallel processor 104.

[0036] CPU 102 includes (not shown) one or more of the following: a control processor, a field - programmable gate array (FPGA), an application - specific integrated circuit (ASIC), or a digital signal processor (DSP). CPU 102 executes at least a portion of the control logic that controls the operation of processing system 100. For example, in various embodiments, CPU 102 executes operating system 108, one or more applications 114, and device driver 116. In some embodiments, CPU 102 initiates and controls the execution of one or more applications 114 by distributing the processing associated with one or more applications 114 across CPU 102 and other processing resources, such as parallel processor 104.

[0037] The parallel processor 104 executes commands and programs for selected functions, such as graphics operations and other operations that may be particularly suitable for parallel processing. The parallel processor 104 is a processor capable of executing a single instruction on multiple data or threads in parallel. Examples of parallel processors include processors such as graphics processing units (GPUs), massively parallel processors, single instruction multiple data (SIMD) architecture processors, and single instruction multiple thread (SIMT) architecture processors for performing graphics, machine intelligence, or computing operations. In some specific implementations, the parallel processor is a separate device included as part of a computer. In other specific implementations such as advanced processor units, the parallel processor is included in a single device together with a host processor such as a central processing unit (CPU). Generally, the parallel processor 104 is commonly used to perform graphics pipeline operations, such as pixel operations, geometric calculations, and rendering images to a display. In some embodiments, the parallel processor 104 also performs computational processing operations (e.g., operations unrelated to graphics, such as video operations, physical simulations, computational fluid dynamics, etc.) based on commands received from the CPU 102. The commands may be executed by a dedicated processor such as a dispatch processor, a command processor, or a network controller.

[0038] In various embodiments, the parallel processor 104 includes one or more computing units 110 as processor cores, and these processor cores include one or more SIMD units (not shown), and the one or more SIMD units execute threads concurrently with the execution of other threads in a wavefront, for example, according to the single instruction multiple data (SIMD) execution model. The SIMD execution model is a model in which multiple processing elements such as arithmetic logic units (ALUs) share a single program control flow unit and a program counter, and thus execute the same program, but are able to execute the program with different data. Some embodiments of the parallel processor 104 are used to implement a GPU, and in this case, the computing unit 110 is referred to as a shader core or a streaming multiprocessor (SMX). The number of computing units 110 implemented in the parallel processor 104 is a matter of design choice. The application 114 executed at one or more of the computing units 110 is referred to as a software client.

[0039] The parallel processor 104 also includes one or more caches 112 configured to store a subset of the data stored at the system memory 106. If the data required by the parallel processor 104 is not found in the cache 112, the cache controller (not shown) issues a memory access request to the memory controller (not shown) to retrieve the data from the system memory 106.

[0040] To facilitate the retirement of memory pages in response to a failure spanning more than one memory page, processing system 100 includes a scrubber 122 and a page retirement policy 124. In some embodiments, scrubber 122 is a hardware scrubber that periodically "scrubs" system memory 106 by reading each location in system memory 106 and corrects hidden (unaccessed) correctable errors before a second failure generates an uncorrectable error. In other embodiments, scrubber 122 is implemented in software and is included, for example, in operating system 108 or device driver 116. Scrubber 122 uses an error correction code (ECC) to correct correctable errors, such as single-bit errors. The time it takes for scrubber 122 to cycle through each location in system memory 106 is referred to as the scrub interval. During the process of scrubbing system memory 106, scrubber 122 also encounters DUEs, such as multi-bit errors. For DUEs, scrubber 122 generates a list of pages to be retired to isolate uncorrected memory errors.

[0041] Page retirement policy 124 proactively (i.e., outside of the scrub interval or based on memory accesses) identifies additional pages of the DRAM for potential retirement in response to detecting an uncorrectable error at a first page of the DRAM. Page retirement policy 124 specifies identifying a second page of the DRAM for potential retirement based on one or more of the following: physical proximity to the first page; being included in an address range stored at a fault map that tracks addresses of DRAM pages with detected faults; and predicting a set of pages to be scrubbed based on a miss at TLB 120.

[0042] Figure 2 is a block diagram of a memory subsystem 200 that includes control logic circuitry 210 according to some embodiments, the control logic circuitry being configured to implement page retirement policy 124 in conjunction with Figure 1 scrubber 122 at system memory 106 of processing system 100. Memory subsystem 200 includes memory modules for storing data to be accessed by instructions executed at CPU 102 and parallel processor 104. Memory subsystem 200 is organized as a memory hierarchy that has system memory 106 at the top of the hierarchy to store all data accessible by the executing instructions, and one or more caches (such as Figure 1 cache 112 as shown) at lower levels of the memory hierarchy to store a subset of the data stored at system memory 106.

[0043] If the data required by the CPU 102 or the parallel processor 104 is not found at the cache 112 of the memory hierarchy, the cache controller (not shown) issues a memory access request to the memory controller 212 to retrieve the data from the system memory 106. In the depicted example, the memory subsystem 200 includes a memory controller 212 and a DIMM 215 that includes two DRAM ranks, namely rank-0 220 and rank-1 230. Rank-0 220 includes DRAM-0 221, DRAM-1 222, DRAM-2 223, and DRAM-N 224. Rank-1 230 includes DRAM-0 231, DRAM-1 232, DRAM-2 233, and DRAM-N 234. Each DRAM includes a plurality of banks that can be accessed in parallel, such as bank-N 240, which in turn includes rows and columns of memory cells, where each row / column address pair uniquely identifies a 4-bit word in the DRAM. In some embodiments, the memory controller 212 controls more than two DRAM ranks.

[0044] In the illustrated example, the memory subsystem 200 is protected by an error detection and correction code (ECC) that stores a number of additional parity bits (not shown) along with each data word. The parity bits are encoded to allow detection of certain errors in the data bits and the parity bits. For some errors, the ECC identifies the specific data bit in the error, thus allowing the scrubber 122 to correct the data. ECC detection and correction are performed for each read access to the DRAM and during erasure by the scrubber 122.

[0045] In addition, to facilitate enhanced page exits for multi-page faults, the page exit policy 124 employs control logic circuitry 210 to proactively identify additional pages of the DRAM for potential exit in response to the scrubber 122 or the memory controller 212 encountering a DUE at the first page of the DRAM. In some embodiments, the control logic circuitry 210 is included in the IOMMU 118 or the device driver 116.

[0046] Figure 3FIG. 300 is a block diagram of a page exit strategy 124 for proactively identifying pages of a memory for eviction based on faults detected in one or more other pages of the memory. In the illustrated example, page exit strategy 124 includes three instruction sets for proactively identifying pages of the memory for eviction. Instruction set 310 specifies checking pages within a physical distance from the page identified as containing a DUE. Thus, in response to encountering a DUE at a first page, instruction set 310 causes the scrubber 122 to erase memory pages within a predetermined distance from the first page via command control logic 210 to determine whether the memory pages also contain a DUE to identify additional DRAM pages for potential eviction.

[0047] In some embodiments, the scrubber 122 or the memory controller 212 encounters a DUE at a first page X. The scrubber 122 or the memory controller 212 causes the first page X to be evicted and swapped with a page H by transferring the data stored at the first page X to the page H. The control logic 210 then identifies adjacent pages Q-W and Y-Z in the physical DRAM space to the first page X and causes the scrubber 122 to erase pages Q-W and Y-Z. In response to encountering a DUE at any of pages Q-W and Y-Z, the scrubber 122 causes the corresponding page to be evicted and swapped in with a corresponding page from the range of pages A-J. If the scrubber 122 does not encounter a DUE at any of pages Q-W and Y-Z, then in some embodiments, the page eraser copies the data from pages Q-W and Y-Z to pages A-G and I-J. Instruction set 310 of page exit strategy 124 specifies the size of the range Q-Z, which can be arbitrarily large or small based on the granularity of the DRAM faults (e.g., bank faults, row faults, etc.) that page exit strategy 124 is programmed to address.

[0048] Instruction set 312 specifies checking a fault map constructed by the scrubber 122 to identify additional DRAM pages for potential eviction. The scrubber 122 checks for errors across the memory subsystem 200 during an erase interval. In response to detecting a DUE at a particular page address, the scrubber 122 records the page address in a data structure (not shown) such as a hash table (referred to herein as a fault map). In some embodiments, the scrubber 122 updates the fault map at each erase interval such that any newly observed faults are recorded and any pages that have been evicted (i.e., taken offline) can be removed from the fault map.

[0049] In some embodiments, instruction set 312 specifies that in response to encountering a DUE at first page X, the scavenger eraser 122 will consult the failure map to determine whether any neighbors of the first page X (i.e., pages having physical addresses within a specified distance of the first page X) have been recorded in the failure map. In some embodiments, the scavenger eraser 122 checks the neighbors of the first page X by scanning the failure map to look for adjacent page addresses. In other embodiments, the scavenger eraser 122 organizes the failure map to track the start and end of adjacent pages that are found to have a DUE during the erase interval. If the scavenger eraser 122 finds one or more neighbors of the first page X in the failure map, the scavenger eraser 122 will swap the adjacent pages with the pages reserved for replacement of the exiting pages.

[0050] Instruction set 314 specifies identifying additional DRAM pages for potential eviction based on a TLB 120 miss. Instruction set 314 predicts which pages are likely to miss in the cache and will thus likely be accessed from the system memory 106 based on a miss at the TLB 120. Instruction set 314 commands the control logic circuit 210 to cause the scavenger eraser 122 to erase the pages predicted to miss. If it is determined that a page predicted to miss in the cache 112 has a DUE, the control logic circuit 210 causes the page to be evicted because the page is predicted to be accessed again. In some embodiments, instruction set 314 commands the control logic circuit 210 to identify multiple adjacent pages that are part of a "miss set". If the scavenger eraser 122 encounters a DUE on one of the pages in the miss set, the control logic circuit 210 causes all of the pages in the miss set to be evicted.

[0051] The page eviction policy 124 determines which instruction sets 310, 312, 314 to use based on the implementation of the processing system 100 and the reliability, availability, and serviceability (RAS) requirements. For example, in some embodiments, instruction set 310 immediately evicts pages with transient or intermittent failures, while instruction set 312 does not evict. The failure map method specified by instruction set 312 allows flexibility in choosing how many pages to evict based on the address ranges tracked in the failure map, while the fixed policy of instruction set 310 does not.

[0052] Figure 4FIG. 400 is a flow chart illustrating a method for identifying a failure at a subset of pages of a DRAM based on detecting an uncorrectable error at a first page of the DRAM according to some embodiments. At block 402, a page exit policy 124 sets a physical distance (distance parameter) from the page found to have a DUE for pages considered to be "proximals" of the page found to have a DUE. At block 404, a scrubber 122 or a memory controller 212 detects an uncorrectable error at a first page of the DRAM. At block 406, a control logic circuit 210 identifies a subset of pages of the DRAM at the physical distance set by the page exit policy 124. At block 408, the control logic circuit 210 causes the scrubber 122 to check for a failure in the subset of pages of the DRAM.

[0053] At block 410, the scrubber 122 determines whether a DUE exists at any of the pages in the subset of pages. If, at block 410, the scrubber 122 determines that a DUE exists at one or more of the pages in the subset, the method flow proceeds to block 412. At block 412, the scrubber 122 causes the affected page to exit and swaps in a replacement page from the DRAM. If, at block 410, the scrubber 122 determines that a DUE does not exist at one or more of the pages in the subset, the method flow proceeds to block 414. At block 414, the scrubber 122 copies data from the subset of pages to a replacement page.

[0054] Figure 5 FIG. 500 is a block diagram of a scrubber 122 according to some embodiments, which identifies a failure at a subset of pages of a DRAM based on a failure map 510 storing addresses of pages with detected uncorrectable errors, such as pages at a bank 240 of a DRAM-N 234. According to an instruction set 312 of a page exit policy 124, a control logic circuit 210 instantiates the scrubber 122 to maintain the failure map 510, which includes addresses of pages determined to contain correctable or uncorrectable failures. In response to encountering a DUE at a page 502 of a DRAM-0 231, the scrubber 122 stores an address 506 of the page 502 containing the DUE at the failure map 510.

[0055] The patrol eraser 122 further consults the failure map 510 to determine whether any neighbors (such as page 504 having a physical address within a predetermined distance of page 502) have been recorded in the failure map 510. In some embodiments, the predetermined distance is set by the page eviction policy 124. In some embodiments, the patrol eraser 122 checks the neighbors of page 502 by scanning the failure map 510 to look for adjacent page addresses. In other embodiments, the failure map 510 is organized to track the start and end of the address ranges of adjacent pages that are found to have a DUE during an erase interval. If the patrol eraser 122 finds one or more neighbors of page 502 at the failure map 510, the patrol eraser 122 swaps the adjacent pages with the pages reserved for replacement of the evicted pages.

[0056] Figure 6 FIG. 600 is a flow chart illustrating a method 600 for identifying failures at a subset of pages of a DRAM based on a failure map 510 storing the addresses of pages having detected uncorrectable errors. At block 602, the page eviction policy 124 sets the area for the patrol eraser 122 to erase and the erase interval. At block 604, the patrol eraser 122 detects a DUE at page 502 of the DRAM. At block 606, the patrol eraser 122 records the address of page 502 at the failure map 510.

[0057] At block 608, the patrol eraser 122 checks the failure map 510 for neighbors of page 502. Based on the page eviction policy 124, neighbors are identified based on having an address within a predetermined distance of page 502. If, at block 610, the patrol eraser 122 finds neighbors of page 502 recorded at the failure map 510, the method flow proceeds to block 612. At block 612, the patrol eraser 122 swaps the adjacent pages with the pages reserved for replacement of the evicted pages. If, at block 610, the patrol eraser 122 does not find neighbors of page 502 at the failure map 510, the method flow proceeds back to block 604.

[0058] Figure 7 FIG. 700 is a block diagram of a control logic circuit 210 for identifying a subset of pages of a DRAM for erasure based on misses at the TLB 120 according to some embodiments. As explained above, the TLB 120 stores a subset of the virtual-to-physical address translations stored at the page table 128. A miss at the TLB 120 predicts a future miss at the cache 112, which will require access to the system memory 106. According to the instruction set 314, the page eviction policy 124 commands the control logic circuit 210 to identify a subset of DRAM pages for erasure based on misses at the TLB 120.

[0059] The control logic circuit 210 identifies cache sets referred to as "miss sets" that are expected to have a large number of misses in the near future, and this large number of misses is expected to cause future accesses to the system memory 106. In some embodiments, as explained below, any page with the same "set_index + offset" bit value is considered to be part of a "miss set".

[0060] Since a miss in the TLB 120 is more likely to result in a miss in the cache 112, in response to a miss in the TLB 120, the control logic circuit 210 begins to determine future misses in the cache sets of the cache 112 before completing the address translation process triggered by the miss at the TLB 120. To predict future misses in the cache sets, the control logic circuit 210 first determines the indices of the sets that are likely to have misses in the near future. In some embodiments, in response to a miss in the TLB 120, the control logic circuit 210 uses a hash function to decode the page offset of the virtual address and then returns the index of the set. Once the hash function generates the set number 706, the control logic circuit 210 inserts the set number 706 into the table 704 of the TLB miss sets. The table 704 of the TLB miss sets tracks the sets referenced by TLB misses. The table 704 of the TLB miss sets includes the set number 706 (once it has been inserted by the control logic circuit 210) and the miss counter 708. In some embodiments, the miss counter 708 is a saturating counter that is incremented whenever there is a miss in the set. If the value of the miss counter 708 is higher than the threshold 710 for a given set, then that set is considered a "miss set" 712. In some embodiments, the threshold 710 is defined by the programmer.

[0061] In some embodiments, the functions of the control logic circuit are performed by the IOMMU 118. Once the IOMMU 118 receives a TLB miss, the IOMMU 118 uses logic to find the set number referenced by the TLB miss. Then, the IOMMU 118 inserts the set number into the table 704 of the TLB miss sets and updates the miss counter 708 corresponding to that set. In some embodiments, the total number of bits for the miss counter 708 is obtained using the logarithm based on the associativity of the cache 112.

[0062] Because only a limited number of sets will be accessed during a particular time interval, the table 704 of TLB miss sets does not need to store information for all sets, and the size of the table 704 of TLB miss sets can be limited by storing only the active sets within the cache 112. In some embodiments, the table 704 of TLB miss sets stores only a total number of sets divided by a number N, and uses a direct mapped cache structure to evict the least recently used line from the table 704 of TLB miss sets, where N is a constant and is defined by the programmer. For example, if the cache 112 is a 2MB cache with an associativity of 16 and a block size of 64B, there will be 2048 sets (11 bits are used to store the set index in the table 704 of TLB miss sets). The miss counter 708 has 4 bits. By assuming N = 4 (such that 1 / 4 of these sets are stored in the table 704 of TLB miss sets), the size of the table 704 of TLB miss sets will be only 0.94KB, which is approximately 0.046% of the size of the cache 112.

[0063] Figures 8 to 10 Illustrates steps for extracting a set index from a virtual address. Figure 8 Is a diagram 800 illustrating the translation of a virtual address 802 to a physical address 812 for identifying a subset of pages of DRAM for erasure based on a miss at the TLB 120 according to some embodiments. The control logic circuit 210 uses a hash function to determine the cache set number before completing the address translation process. As Figure 8 Shown, during the translation of the virtual address 802 to the physical address 812, only the virtual page number 804 is translated to the physical page number 814, and there is no translation of the virtual page offset 806 to the physical page offset 816 because the page offsets 806, 816 are the same for both the virtual address 802 and the physical address 812. The hash function uses only the page offset portion 806 of the virtual address 802 to determine the set number. Generally speaking, the page offset is k bits, which is the logarithm (page size).

[0064] Figure 9 Is a diagram 900 illustrating the extraction of a set index 904 from the virtual page offset 806 for identifying a subset of pages of DRAM for erasure based on a miss at the TLB 120 according to some embodiments. As Figure 9 Shown, in each cache physical address, a, b, and c bits are used for byte offset 906, set index 904, and tag 902 respectively. If a + b is less than or equal to k, the control logic circuit 210 can determine the exact number of sets. For small-sized caches or caches with higher associativity and fewer set numbers, the value of k is typically equal to or higher than a + b. However, for those cases where the value of a + b is higher than k, the hash function will return a set of set indices.

[0065] Figure 10 FIG. 1000 is an illustration of an example of a set index according to some embodiments, where the number of page offset bits is less than the total number of bits required for a set index and an offset to identify a subset of pages of a DRAM for erasure based on a miss at a translation lookaside buffer. For example, if (a + b) – k = 2, as Figure 10 shown, there are two bits with unknown values until the virtual-to-physical address translation is complete. In this case, the control logic circuit 210 considers all four possibilities of the set number, inserts all four possibilities into the table 704 of the TLB miss set, and increments its corresponding miss counter 708. Once the pages in the miss set 712 have been identified, the control logic circuit 210 dispatches the task of erasing the pages in the miss set 712 to the patrol eraser 122 to determine whether these pages contain any defects. If the patrol eraser 122 encounters a DUE at any of the pages in the miss set 712, the patrol eraser 122 retires all the pages in the miss set 712.

[0066] Figure 11 FIG. 1100 is a flow chart of a method for identifying a subset of pages of a DRAM for erasure based on a miss at the TLB 120 according to some embodiments. At block 1102, the control logic circuit 210 identifies a miss at the TLB 120. At block 1104, the control logic circuit 210 decodes the page offset of the virtual address 802 of the miss and returns the index of the set. At block 1106, the control logic circuit 210 inserts the set number 706 into the table 704 of the TLB miss set, and at block 1108, the control logic circuit 210 increments the miss counter 708 for the miss set 712. At block 1110, the control logic circuit 210 determines whether the miss counter 708 for the miss set 712 exceeds a threshold 710. If at block 1110, the control logic circuit 210 determines that the miss counter 708 does not exceed the threshold 710, the method flow continues back to block 1102.

[0067] If at block 1110, the control logic circuit 210 determines that the miss counter 708 exceeds the threshold 710, the method flow continues to block 1112. At block 1112, the patrol eraser 122 erases the pages in the miss set 712. At block 1114, the patrol eraser 122 determines whether there is a DUE at any of the pages in the miss set 712. If at block 1114, the patrol eraser 122 determines that there is no DUE at one or more of the pages in the miss set 712, the method flow continues back to block 1102.

[0068] If, at block 1114, the patrol eraser 122 determines that there is a DUE at one or more of the pages in the miss set 712, the method flow proceeds to block 1116. At block 1116, the patrol eraser 122 evicts all of the pages in the miss set 712.

[0069] In some embodiments, the above-described apparatus and techniques are implemented in a system that includes one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips), such as the processing system described above with reference to Figures 1 to 11 The processing system described above. Electronic design automation (EDA) and computer-aided design (CAD) software tools may be used in the design and manufacture of these IC devices. These design tools are typically represented as one or more software programs. One or more software programs include code executable by a computer system to manipulate the computer system to operate on code representative of a circuit of one or more IC devices in order to perform at least a portion of a process for designing or tuning a manufacturing system to manufacture the circuit. The code may include instructions, data, or a combination of instructions and data. Software instructions representative of design tools or manufacturing tools are typically stored in a computer-readable storage medium accessible to a computing system. Similarly, code representative of one or more stages of the design or manufacture of an IC device may be stored in and accessed from the same computer-readable storage medium or different computer-readable storage media.

[0070] A computer-readable storage medium may include any non-transitory storage medium or combination of non-transitory storage media that is accessible to a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard disk drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical systems (MEMS)-based storage media. A computer-readable storage medium may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to the computing system (e.g., magnetic hard disk drive), removably attached to the computing system (e.g., optical disc or universal serial bus (USB)-based flash memory), or coupled to the computing system via a wired or wireless network (e.g., network-attached storage (NAS)).

[0071] In some embodiments, certain aspects of the above techniques may be implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions that are stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions and certain data that, when executed by one or more processors, manipulate the one or more processors to perform one or more aspects of the above techniques. The non-transitory computer-readable storage medium may include, for example, a disk or optical storage device, a solid-state storage device such as flash memory, a cache, a random access memory (RAM), or other one or more non-volatile memory devices, etc. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executed by one or more processors.

[0072] Note that not all activities or elements described above in the general description are required, that a portion of a particular activity or device may not be required, and that one or more additional activities may be performed or elements may be included in addition to those described. Further, the order of listing activities is not necessarily the order in which they are performed. Additionally, these concepts have been described with reference to specific embodiments. However, those of ordinary skill in the art understand that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the following claims. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the present disclosure.

[0073] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature that may cause any benefit, advantage, or solution to occur or become more pronounced should not be construed as a critical, required, or essential feature of any or all of the claims. Further, the specific embodiments disclosed above are illustrative only, as the disclosed subject matter may be modified and practiced in different but equivalent manners obvious to those of ordinary skill in the art that benefit from the teachings herein. No limitation with respect to the details of the construction or design shown herein is intended other than as described in the following claims. It is, therefore, evident that the specific embodiments disclosed above may be altered or modified, and all such variations are considered to be within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the following claims.

Claims

1. A method, the method comprising: In response to detecting an uncorrectable error at a first page of a dynamic random access memory (DRAM), identifying a second page of the DRAM for potential retirement.

2. The method according to claim 1, wherein identifying the second page comprises: Identifying a subset of pages of the DRAM at a first physical distance from the first page, the second page being included in the subset of pages; And Checking for the presence of a fault at the subset of pages.

3. The method according to claim 2, the method further comprising: Retiring the second page in response to detecting an uncorrectable error at the second page.

4. The method according to any one of claims 1 to 3, wherein detecting the uncorrectable error at the first page is performed while checking the DRAM for faults.

5. The method according to claim 4, the method further comprising: Recording the uncorrectable error detected at the first page at a fault map including a plurality of page addresses of pages having faults.

6. The method according to claim 5, wherein identifying the second page comprises: Scanning the fault map for page addresses adjacent to the first page in response to recording the uncorrectable error at the first page at the fault map.

7. The method according to claim 6, the method further comprising: Selecting a subset of page addresses for retirement based on the presence of an uncorrectable error at one or more pages in the subset, the subset including the page address of the second page.

8. The method according to claim 1, wherein identifying the second page comprises: Identifying a subset of pages of the DRAM to be accessed predicted based on a miss at a translation lookaside buffer (TLB), the second page being included in the subset of pages; And Retiring the subset of pages in response to detecting an uncorrectable error at the second page.

9. A method, the method comprising: Retiring a first page of the DRAM before accessing the first page in response to detecting an uncorrectable error at a second page of the dynamic random access memory (DRAM).

10. The method according to claim 9, the method further comprising: Identifying a subset of pages of the DRAM at a first physical distance from the second page, the first page being included in the subset of pages; And Checking for the presence of a fault at the subset of pages.

11. The method according to claim 10, the method further comprising: Retiring the first page in response to detecting an uncorrectable error at the second page.

12. The method according to any one of claims 9 to 11, the method further comprising: Detecting the uncorrectable error at the first page while checking the DRAM for faults.

13. The method according to claim 12, the method further comprising: Recording the uncorrectable error detected at the first page at a fault map including a plurality of page addresses of pages having faults.

14. The method according to claim 13, the method further comprising: Scanning the fault map for page addresses adjacent to the second page in response to recording the uncorrectable error at the second page in the fault map.

15. The method according to claim 14, the method further comprising: Selecting a subset of page addresses for eviction based on the presence of uncorrectable errors at one or more pages in the subset, the subset including the page address of the first page.

16. The method according to claim 9, the method further comprising: Identifying a subset of pages of the DRAM to be accessed predicted based on a miss at a translation lookaside buffer (TLB), the first page being included in the subset of pages; And Evicting the subset of pages in response to detecting an uncorrectable error at the first page.

17. An apparatus, the apparatus comprising: Dynamic random access memory (DRAM); And Control logic circuitry configured to: In response to detecting an uncorrectable error at a first page of the DRAM, identify a second page of the DRAM for potential eviction.

18. The apparatus according to claim 17, wherein the control logic circuitry is further configured to: Identify a subset of pages of the DRAM at a first physical distance from the first page, the second page being included in the subset of pages; and Check for the presence of faults at the subset of pages.

19. The apparatus according to claim 17, wherein the control logic circuitry is further configured to: Record the uncorrectable error detected at the first page at a fault map including a plurality of page addresses of pages with faults; Scan the fault map for page addresses adjacent to the second page in response to recording the uncorrectable error at the second page in the fault map; and Select a subset of page addresses for eviction based on the presence of uncorrectable errors at one or more pages in the subset, the subset including the page address of the first page.

20. The apparatus according to claim 17, wherein the control logic circuitry is further configured to: Identify a subset of pages of the DRAM to be accessed predicted based on a miss at a translation lookaside buffer (TLB), the second page being included in the subset of pages; and Evict the subset of pages in response to detecting an uncorrectable error at the second page.

Images