Page Retirement Techniques for Multi-Page DRAM Faults

Abstract

A processing system (100) uses techniques for enhancing dynamic random access memory (DRAM) page retirement to facilitate identification and retirement of pages affected by multi-page DRAM faults. In response to detecting an uncorrectable error in a first page of the DRAM, the processing system identifies a second page of the DRAM for potential retirement based on one or more of physical proximity to the first page, inclusion within a range of addresses stored in a fault map (510) that tracks addresses of DRAM pages with detected faults, and predicting a set of pages to check for faults based on misses in a translation lookaside buffer (TLB) (120).

Description

[Background technology]

[0001] Computer memory, such as dynamic random access memory (DRAM), is susceptible to a variety of transient and permanent faults. Transient faults cause incorrect data to be read from a memory location until the memory location is overwritten with correct data and do not indicate device damage, while permanent faults (i.e., hard faults) cause a memory location to consistently return incorrect values. In response to encountering a permanent fault, called a detectable uncorrectable error (DUE), a processing system retires the affected portion of memory. Summary of the Invention [Means for solving the problem]

[0002] In examples described herein, techniques are provided for enhancing DRAM page retirement in a processing system to facilitate identification and retirement of pages affected by multi-page DRAM failures. In one exemplary embodiment, a method includes identifying a second page of dynamic random access memory (DRAM) for potential retirement in response to detecting an uncorrectable error in a first page of the DRAM.

[0003] In some embodiments, identifying the second page includes identifying a subset of pages of the DRAM that are a first physical distance from the first page, the second page being included in the subset of pages. The identifying further includes checking for the presence of faults in the subset of pages. In some embodiments, the method further includes retiring the second page in response to detecting an uncorrectable error in the second page. Detecting the uncorrectable error in the first page may be performed while checking for faults in the DRAM.

[0004] In some embodiments, the method further includes recording the uncorrectable error detected in the first page in a fault map including a plurality of page addresses of the failing pages. Identifying the second page may include scanning the fault map for page addresses adjacent to the first page in response to recording the uncorrectable error in the first page in the fault map. In some embodiments, the method further includes selecting a subset of page addresses for retirement based on the presence of the uncorrectable error in one or more pages of the subset, the subset including the page address of the second page. Identifying the second page may include identifying a subset of pages of the DRAM predicted to be accessed based on a miss in a translation lookaside buffer (TLB), the second page being included in the subset of pages, and retiring the subset of pages in response to detecting the uncorrectable error in the second page.

[0005] In another example, a method includes retiring a first page of dynamic random access memory (DRAM) before accessing the first page in response to detecting an uncorrectable error in a second page of the DRAM. In some embodiments, the method further includes identifying a subset of pages of the DRAM that are a first physical distance from the second page, where the first page is included in the subset of pages, and checking for the presence of faults in the subset of pages. The method may further include retiring the first page in response to detecting the uncorrectable error in the second page.

[0006] In some embodiments, the method further includes detecting an uncorrectable error in the first page while checking the DRAM for faults. The method may also include recording the uncorrectable error detected in the first page in a fault map that includes a plurality of page addresses of pages having the fault. In some embodiments, the method further includes, in response to recording the uncorrectable error in the second page in the fault map, scanning the fault map for page addresses adjacent to the second page.

[0007] The method may include selecting a subset of page addresses for retirement based on the presence of an uncorrectable error in one or more pages of the subset, the subset including the page address of the first page. In some embodiments, the method further includes identifying a subset of pages of the DRAM predicted to be accessed based on a miss in a translation lookaside buffer (TLB), the first page being included in the subset of pages, and retiring the subset of pages in response to detecting an uncorrectable error in the first page.

[0008] In another example, a device includes a dynamic random access memory (DRAM) and control logic configured to identify a second page of the DRAM for potential retirement in response to detecting an uncorrectable error in a first page of the DRAM. In some embodiments, the control logic is further configured to identify a subset of pages of the DRAM that are a first physical distance from the first page, where the second page is included in the subset of pages, and check for the presence of faults in the subset of pages.

[0009] The control logic may be further configured to record the uncorrectable error detected in the first page in a fault map including multiple page addresses of the faulty pages, and in response to recording the uncorrectable error in the second page in the fault map, scan the fault map for page addresses adjacent to the second page. The control logic may also be configured to select a subset of page addresses for retirement based on the presence of an uncorrectable error in one or more pages of the subset, the subset including the page address of the first page. In some embodiments, the control logic is configured to identify a subset of pages of the DRAM predicted to be accessed based on a miss in a translation lookaside buffer (TLB), where the second page is included in the subset of pages, and in response to detecting the uncorrectable error in the second page, retire the subset of pages.

[0010] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which: The use of the same reference numbers in different drawings indicates similar or identical items. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram of a processing system configured to implement a page retirement policy for preemptively identifying pages of memory for retirement based on detected faults in other pages of the memory, according to some embodiments. [Figure 2] FIG. 1 is a block diagram of control logic configured to implement a page retirement policy in conjunction with a patrol scrubber in a system memory of a processing system, according to some embodiments. [Figure 3] FIG. 1 is a block diagram of a page retirement policy for preemptively identifying pages of memory for retirement based on detected faults in other pages of the memory, according to some embodiments. [Figure 4] FIG. 1 is a flow diagram illustrating a method for identifying failures in a subset of pages of a DRAM based on detecting an uncorrectable error in a first page of the DRAM, according to some embodiments. [Figure 5] FIG. 1 is a block diagram of a patrol scrubber that identifies failures in a subset of pages of a DRAM based on a failure map that stores addresses of pages in which uncorrectable errors are detected, according to some embodiments. [Figure 6] FIG. 1 is a flow diagram illustrating a method for identifying failures in a subset of pages of a DRAM based on a failure map that stores addresses of pages in which uncorrectable errors are detected, according to some embodiments. [Figure 7] FIG. 1 is a block diagram of control logic that identifies a subset of pages of a DRAM to scrub based on misses in a translation lookaside buffer, according to some embodiments. [Figure 8] FIG. 1 illustrates the translation of virtual addresses to physical addresses to identify a subset of pages of a DRAM for scrubbing based on a miss in a translation lookaside buffer, according to some embodiments. [Figure 9] FIG. 10 illustrates the extraction of a set index from a virtual page offset that identifies a subset of pages of DRAM to scrub based on a miss in the translation lookaside buffer, according to some embodiments. [Figure 10] FIG. 10 illustrates an example of a set index with a number of page offset bits that is less than the total number of bits required for the set index and offset to identify a subset of pages of DRAM for scrubbing based on a miss in the translation lookaside buffer, according to some embodiments. [Figure 11] FIG. 1 is a flow diagram illustrating a method for identifying a subset of pages of a DRAM to scrub based on a miss in a translation lookaside buffer, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0012] Conventional processing systems typically use one or more memory modules (e.g., industry-standard memory such as dual in-line memory modules (DIMMs)) having an array of memory cells. DIMMs are composed of rows of DRAM chips, which are organized into banks of rows and columns of memory cells configured to store data. Conventional retirement algorithms retire portions of memory affected by a DUE at the granularity level of a single page (i.e., 4 kilobytes), which typically maps to a single DRAM row. A page is a fixed-length contiguous block of virtual memory described by a single entry in a page table. A memory page is the smallest unit of data for memory management in a virtual memory operating system.

[0013] However, larger granularity faults in DRAM can affect multiple rows or entire DRAM banks. Based on how data is interleaved in the memory system, a single-page retirement action (whether initiated in response to a DUE or a pattern of corrected errors) typically retires a single row or part of a DRAM row. Thus, a DUE that spans multiple pages of memory, such as a multi-bit fault in a high-bandwidth memory (HBM2 / 2E) or an infinite fault in HBM3, can require multiple invocations of a processing system driver to initiate page retirement, adversely affecting system availability and performance.

[0014] Processing systems often use virtual memory to handle data accesses by executing programs (e.g., applications, operating systems, device drivers, etc.). In such processing systems, instead of accessing memory using addresses based on the physical location of pages of memory (or "physical addresses"), the programs access memory using "virtual addresses" within a "virtual address space," which is a local address space specific to the corresponding program. Therefore, to support memory accesses, processing systems typically use address translation circuitry to translate virtual addresses into corresponding physical addresses. The address translation circuitry uses one or more translation lookaside buffers (TLBs) to cache virtual-to-physical address translations for efficient lookup by the processor core.

[0015] 1-11 illustrate techniques for enhancing DRAM page retirement in a processing system to facilitate identification and retirement of pages affected by a multi-page DRAM fault. In response to detecting an uncorrectable error in a first page of DRAM, the processing system identifies one or more additional pages of DRAM (collectively referred to herein as "second pages") for potential retirement based on one or more of: physical proximity to the first page; inclusion in a range of addresses stored in a fault map that tracks addresses of DRAM pages with detected faults; and predicting a set of pages to check for faults based on misses in the TLB. By identifying the additional pages of DRAM for potential retirement based on the DUE in the first page, the processing system preemptively retires pages affected by the multi-page fault, reducing driver calls to initiate page retirement.

[0016] In some embodiments, the processing system employs a page retirement policy that specifies that, in response to encountering a DUE in a first page of the DRAM, the processing system scrubs (i.e., checks for errors) a subset of pages within a predetermined physical distance from the first page for the presence of a large (multi-page) fault. The predetermined physical distance is set based on the page retirement policy. If the processing system encounters a fault in the subset of pages, the processing system retires the subset of pages.

[0017] In another embodiment, the processing system uses a page retirement policy that specifies that, in response to encountering a DUE in a first page of DRAM, the processing system adds the address of the page with the DUE to a fault map maintained by a patrol scrubber. The patrol scrubber scans the DRAM at intervals for faults and lists the addresses of page faults in the fault map. The processing system determines whether the address of the first page falls within a set of pages identified in the fault map as falling within a major fault. If the address of the first page falls within the range of addresses of the major fault, the processing system retires all pages within that range.

[0018] In some embodiments, the processing system uses a page retirement policy that identifies pages for scrubbing based on misses in the TLB. A TLB miss predicts a miss in the cache that requires a memory access. If the patrol scrubber encounters a DUE in a page included in a subset of contiguous pages predicted to miss in the cache (referred to herein as a "miss set"), the processing system retires the entire subset of contiguous pages. In some embodiments, the processing system identifies a miss set as pages that have the same set index and offset bit value. In response to a miss in the TLB, the processing system uses a hash function to decode the page offset of the virtual address of the TLB miss and return the index of the set. The processing system inserts the set number into a table to track the set referenced by the TLB miss. When the number of misses to a set exceeds a threshold, the patrol scrubber scrubs (i.e., reads and checks for errors in) the pages of the set to identify any DUEs within the set. FIG. 1 is a block diagram of a processing system configured to implement a page retirement policy for preemptively identifying pages of memory for retirement based on detected faults in other pages of memory, according to some embodiments. Processing system 100 includes a central processing unit (CPU) 102 and a parallel processing unit (PPU) 104, also referred to herein as a parallel processor 104. In various embodiments, CPU 102 includes one or more single-core or multi-core CPUs. In various embodiments, parallel processor 104 includes any cooperating collection of hardware and / or software that performs functions and calculations associated with accelerated graphics processing tasks, data-parallel tasks, and nested data-parallel tasks in an accelerated manner in conjunction with resources such as traditional CPUs, traditional graphics processing units (GPUs), and combinations thereof. In the embodiment of FIG. 1, processing system 100 is formed on a single silicon die or package that combines CPU 102 and parallel processor 104 to provide a unified programming and execution environment. This environment allows parallel processor 104 to be used fluidly as CPU 102 for several programming tasks. In other embodiments, CPU 102 and parallel processors 104 are formed separately and mounted on the same or different substrates. It should be understood that processing system 100 may include one or more software, hardware, and firmware components in addition to or different from those shown in FIG. 1 . For example, processing system 100 may further include one or more input interfaces, non-volatile storage, one or more output interfaces, a network interface, and one or more displays or display interfaces. Processing system 100 may include, for example, a server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a game console, etc. As shown in FIG. 1 , processing system 100 includes system memory 106, operating system 108, communications infrastructure 130, and one or more applications 114. Access to system memory 106 is managed by a memory controller (not shown) coupled to system memory 106. For example, requests from CPU 102 or other devices to read from or write to system memory 106 are managed by the memory controller. In some embodiments, one or more applications 114 include various programs or commands for performing computations that are also executed by CPU 102. CPU 102 sends selected commands to parallel processors 104 for processing. Operating system 108 and communications infrastructure 130 are described in more detail below. Processing system 100 further includes a memory management unit, such as device drivers 116 and an input / output memory management unit (IOMMU) 118. Components of processing system 100 may be implemented as hardware, firmware, software, or any combination thereof. In some embodiments, processing system 100 includes one or more software, hardware, and firmware components in addition to or different from those shown in FIG.

[0019] Within processing system 100, system memory 106 includes non-persistent memory such as DRAM (not shown). In various embodiments, system memory 106 stores processing logic instructions, constant values, variable values ​​during execution of an application or portion of other processing logic, or other desired information. For example, in various embodiments, portions of the control logic for executing one or more operations on CPU 102 reside in system memory 106 during execution of the respective portion of the operation by CPU 102. During execution, respective applications, operating system functions, processing logic commands, and system software reside in system memory 106. Control logic commands essential to operating system 108 generally reside in system memory 106 during execution. In some embodiments, other software commands (e.g., device drivers 116) reside in system memory 106 during execution of processing system 100.

[0020] System memory 106 includes a page table 128 that maintains a record of page table entries that store virtual-to-physical address translation information for pages of data stored in the system memory. Upon receiving a request from a program to access memory at a given virtual address, CPU 102 or parallel processor 104 performs a page table walk to obtain corresponding physical address information from page table 128 for the page table entry that provides the physical address associated with the virtual address.

[0021] The IOMMU 118 is a multi-context memory management unit. As used herein, a context is considered the environment in which a kernel executes and the domain in which synchronization and memory management are defined. A context includes a set of devices, memory accessible to those devices, corresponding memory properties, and one or more command queues used to schedule execution of kernels or operations on memory objects. The IOMMU 118 includes logic for performing virtual-to-physical address translations for memory page accesses for devices such as the parallel processors 104. In some embodiments, the IOMMU 118 includes or has access to a translation lookaside buffer (TLB) 120. The TLB 120, illustratively implemented with a content-addressable memory (CAM), accelerates the translation of logical (i.e., virtual) memory addresses to physical memory addresses for requests made by the parallel processors 104 for data in the system memory 106. The TLB 120 stores a subset of the virtual-to-physical address information stored in the page table 128. In some embodiments, TLB 120 is implemented as a hierarchy of multiple TLBs.

[0022] In various embodiments, communications infrastructure 130 interconnects the components of processing system 100. Communications infrastructure 130 includes one or more of a peripheral component interconnect (PCI) bus, an extended PCI (PCI-E) bus, an advanced microcontroller bus architecture (AMBA) bus, an advanced graphics port (AGP), or other communications infrastructure and interconnects (not shown). In some embodiments, communications infrastructure 130 includes an Ethernet network or any other suitable physical communications infrastructure that meets the data transfer rate requirements of the application. Communications infrastructure 130 includes functionality for interconnecting components (including components of processing system 100).

[0023] Drivers, such as device driver 116, communicate with devices (e.g., parallel processor 104) via interconnect or communication infrastructure 130. When a calling program invokes a routine in device driver 116, device driver 116 issues a command to the device. When the device sends data back to device driver 116, device driver 116 invokes the routine of the original calling program. Generally, device drivers are hardware-dependent and operating system-specific to provide the requisite interrupt handling for any necessary asynchronously dependent hardware interfaces. In some embodiments, compiler 126 is embedded within device driver 116. Compiler 126 compiles source code into program instructions necessary for execution by processing system 100. During such compilation, compiler 126 applies transformations to program instructions at various stages of compilation. In other embodiments, compiler 126 is a stand-alone application. In various embodiments, device driver 116 controls the operation of parallel processor 104, for example, by providing an application programming interface (API) to software (e.g., application 114) executing on CPU 102 for accessing various functions of parallel processor 104.

[0024] CPU 102 includes one or more of a control processor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a digital signal processor (DSP) (not shown). CPU 102 executes at least a portion of the control logic that controls the operation of processing system 100. For example, in various embodiments, CPU 102 executes operating system 108, one or more applications 114, and device drivers 116. In some embodiments, CPU 102 initiates and controls the execution of one or more applications 114 by distributing processing associated with one or more applications 114 across CPU 102 and other processing resources, such as parallel processors 104.

[0025] Parallel processor 104 executes commands and programs for selected functions, such as graphics operations and other operations that may be particularly suited to parallel processing. Parallel processor 104 is a processor that can execute a single instruction on multiple data or threads in parallel. Examples of parallel processors include processors such as graphics processing units (GPUs), massively parallel processors, single instruction, multiple data (SIMD) architecture processors, and single instruction, multiple thread (SIMT) architecture processors for performing graphics, machine intelligence, or computational operations. In some embodiments, the parallel processor is a separate device included as part of a computer. In other embodiments, such as an advanced processor unit, the parallel processor is included in a single device along with a host processor, such as a central processor unit (CPU). Parallel processor 104 is typically used to perform graphics pipeline operations, such as pixel operations, geometric calculations, and rendering images to a display. In some embodiments, parallel processor 104 performs computational operations (e.g., operations unrelated to graphics, such as video operations, physics simulations, computational fluid dynamics, etc.) based on commands received from CPU 102. The commands may be executed by a specialized processor such as a dispatch processor, a command processor, or a network controller.

[0026] In various embodiments, the parallel processor 104 includes one or more compute units 110, which are processor cores including one or more SIMD units (not shown) that execute threads concurrently with the execution of other threads within a wavefront, e.g., according to a single instruction, multiple data (SIMD) execution model. A SIMD execution model is one in which multiple processing elements, such as arithmetic logic units (ALUs), share a single program control flow unit and program counter, and thus can execute the same program but on different data. Some embodiments of the parallel processor 104 are used to implement a GPU, in which case the compute units 110 are referred to as shader cores or streaming multiprocessors (SMX). The number of compute units 110 implemented in the parallel processor 104 is a matter of design choice. Applications 114 executing on one or more of the compute units 110 are referred to as software clients.

[0027] Parallel processor 104 also includes one or more caches 112 configured to store a subset of the data stored in system memory 106. If data needed by parallel processor 104 is not found in cache 112, a cache controller (not shown) issues a memory access request to a memory controller (not shown) to retrieve the data from system memory 106.

[0028] To facilitate the retirement of memory pages in response to faults that span multiple memory pages, the processing system 100 includes a patrol scrubber 122 and a page retirement policy 124. In some embodiments, the patrol scrubber 122 is a hardware scrubber that periodically "scrubs" the system memory 106 by reading every location in the system memory 106 and correcting latent (unaccessed) correctable errors before a second fault generates an uncorrectable error. In other embodiments, the patrol scrubber 122 is implemented in software, for example, in the operating system 108 or device driver 116. The patrol scrubber 122 uses an error correction code (ECC) to correct correctable errors, such as single-bit errors. The time it takes the patrol scrubber 122 to cycle through every location in the system memory 106 is referred to as the scrub interval. In the process of scrubbing the system memory 106, the patrol scrubber 122 also encounters DUEs, such as multi-bit errors. In the case of a DUE, the patrol scrubber 122 generates a list of pages to be retired to isolate uncorrected memory errors.

[0029] Page retirement policy 124 preemptively (i.e., outside of a scrub interval or following a memory access) identifies additional pages of DRAM for potential retirement in response to detecting an uncorrectable error in a first page of DRAM. Page retirement policy 124 specifies identification of a second page of DRAM for potential retirement based on one or more of physical proximity to the first page, inclusion in a range of addresses stored in a fault map that tracks addresses of DRAM pages with detected faults, and predicting a set of pages to scrub based on a miss in TLB 120.

[0030] 2 is a block diagram of a memory subsystem 200 including control logic 210 configured to implement a page retirement policy 124 in conjunction with a patrol scrubber 122 in the system memory 106 of the processing system 100 of FIG. 1 , according to some embodiments. The memory subsystem 200 includes memory modules for storing data accessed by instructions executed by the CPU 102 and parallel processors 104. The memory subsystem 200 is organized into a memory hierarchy with the system memory 106 at the top of the hierarchy for storing all data that may be accessed by the executing instructions, and one or more caches (such as cache 112 shown in FIG. 1 ) at lower levels of the memory hierarchy for storing subsets of the data stored in the system memory 106.

[0031] If data needed by CPU 102 or parallel processor 104 is not found in cache 112 of the memory hierarchy, a cache controller (not shown) issues a memory access request to memory controller 212 to retrieve the data from system memory 106. In the illustrated example, memory subsystem 200 includes memory controller 212 and DIMM 215, which includes two ranks of DRAM: rank-0 220 and rank-1 230. Rank-0 220 includes DRAM-0 221, DRAM-1 222, DRAM-2 223, and DRAM-N 224. Rank-1 230 includes DRAM-0 231, DRAM-1 232, DRAM-2 233, and DRAM-N 234. Each DRAM includes multiple banks that can be accessed in parallel, such as bank-N 240, which includes rows and columns of memory cells where each row / column address pair uniquely identifies a 4-bit word within the DRAM. In some embodiments, memory controller 212 controls more than two ranks of DRAM.

[0032] In the illustrated example, memory subsystem 200 is protected by an error detection and correction code (ECC) that stores several additional check bits (not shown) with each data word. The check bits are coded to allow detection of specific errors in the data bits and check bits. For some errors, the ECC identifies the specific data bit in error, allowing patrol scrubber 122 to correct the data. ECC detection and correction is performed on every read access to the DRAM and during scrubbing by patrol scrubber 122.

[0033] Additionally, to facilitate extended page retirement for multi-page faults, page retirement policy 124 uses control logic 210 to preemptively identify additional pages of DRAM for potential retirement in response to patrol scrubber 122 or memory controller 212 encountering a DUE in a first page of DRAM. In some embodiments, control logic 210 is included in IOMMU 118 or device driver 116.

[0034] 3 is a block diagram 300 of a page retirement policy 124 for preemptively identifying pages of memory for retirement based on detected faults in one or more other pages of memory, according to some embodiments. In the illustrated example, page retirement policy 124 includes three instruction sets for preemptively identifying pages of memory for retirement. Instruction set 310 specifies checking pages within a physical distance from a page identified as containing a DUE. Thus, in response to encountering a DUE in a first page, instruction set 310 identifies additional DRAM pages for potential retirement by instructing control logic 210 to cause patrol scrubber 122 to scrub memory pages located within a predetermined distance from the first page to determine whether the memory page also contains a DUE.

[0035] In some embodiments, the patrol scrubber 122 or memory controller 212 encounters a DUE in a first page X. The patrol scrubber 122 or memory controller 212 retires the first page X and swaps it with page H by transferring the data stored in the first page X to page H. The control logic 210 then identifies adjacent pages QW and YZ that are adjacent to the first page X in physical DRAM space and causes the patrol scrubber 122 to scrub pages QW and YZ. In response to encountering a DUE in any of pages QW and YZ, the patrol scrubber 122 retires the corresponding page and swaps in the corresponding page from a range of pages AJ. If the patrol scrubber 122 does not encounter a DUE in any of pages QW and YZ, in some embodiments, the page scrubber copies data from pages QW and YZ to pages AG and IJ. The instruction set 310 of the page retirement policy 124 specifies the size of the range QZ, which can be arbitrarily large or small based on the granularity of the DRAM faults (e.g., bank faults, row faults, etc.) that the page retirement policy 124 is programmed to address.

[0036] The instruction set 312 specifies checking a fault map built by the patrol scrubber 122 to identify additional DRAM pages for potential retirement. The patrol scrubber 122 checks for errors across the memory subsystem 200 during a scrub interval. In response to detecting a DUE at a particular page address, the patrol scrubber 122 records the page address in a data structure (not shown), such as a hash table, referred to herein as a fault map. In some embodiments, the patrol scrubber 122 updates the fault map at each scrub interval so that any new faults observed are recorded and any pages that have already been retired (i.e., taken offline) can be removed from the fault map.

[0037] In some embodiments, instruction set 312 specifies that, in response to encountering a DUE in first page X, patrol scrubber 122 consults the fault map to determine whether any neighboring pages of first page X (i.e., pages with physical addresses within a specified distance of first page X) are recorded in the fault map. In some embodiments, patrol scrubber 122 checks neighboring pages of first page X by scanning the fault map for adjacent page addresses. In other embodiments, patrol scrubber 122 organizes the fault map to track the start and end of neighboring pages found to have a DUE during the scrub interval. If patrol scrubber 122 finds one or more neighboring pages of first page X in the fault map, patrol scrubber 122 swaps the neighboring pages with pages reserved for replacing retired pages.

[0038] The instruction set 314 specifies identifying additional DRAM pages for potential retirement based on misses in the TLB 120. The instruction set 314 predicts which pages are prone to miss in the cache and therefore likely to be accessed from the system memory 106 based on misses in the TLB 120. The instruction set 314 instructs the control logic 210 to cause the patrol scrubber 122 to scrub the pages predicted to miss. If the page predicted to miss in the cache 112 is determined to have a DUE, the control logic 210 retires the page because the page is predicted to be accessed again. In some embodiments, the instruction set 314 instructs the control logic 210 to identify multiple adjacent pages that are part of a "miss set." If the patrol scrubber 122 encounters a DUE on any of the pages in the miss set, the control logic 210 retires all of the pages in the miss set.

[0039] The page retirement policy 124 determines which instruction set 310, 312, 314 to use based on the implementation and reliability, availability, and serviceability (RAS) requirements of the processing system 100. For example, in some embodiments, the instruction set 310 immediately retires pages with transient or intermittent faults, while the instruction set 312 does not. The fault map approach specified by the instruction set 312 allows flexibility in selecting how many pages are retired based on the address ranges tracked in the fault map, while the fixed policy of the instruction set 310 does not.

[0040] 4 is a flow diagram illustrating a method 400 for identifying faults in a subset of pages of the DRAM based on detecting an uncorrectable error in a first page of the DRAM, according to some embodiments. At block 402, the page retirement policy 124 sets a physical distance (distance parameter) from the page found to have the DUE for pages that are considered "neighbors" of the page found to have the DUE. At block 404, the patrol scrubber 122 or the memory controller 212 detects an uncorrectable error in the first page of the DRAM. At block 406, the control logic 210 identifies a subset of pages of the DRAM that are within the physical distance set by the page retirement policy 124. At block 408, the control logic 210 causes the patrol scrubber 122 to check the subset of pages of the DRAM for the presence of faults.

[0041] At block 410, the patrol scrubber 122 determines whether any of the pages in the subset of pages have a DUE. If at block 410, the patrol scrubber 122 determines that one or more of the pages in the subset have a DUE, method flow proceeds to block 412. At block 412, the patrol scrubber 122 retires the affected page and swaps in a replacement page from DRAM. If at block 410, the patrol scrubber 122 determines that one or more of the pages in the subset do not have a DUE, method flow proceeds to block 414. At block 414, the patrol scrubber 122 copies data from the subset of pages to the replacement page.

[0042] 5 is a block diagram 500 of patrol scrubber 122 that identifies faults in a subset of pages of a DRAM, such as pages in bank 240 of DRAM-N 234, based on a fault map 510 that stores addresses of pages in which uncorrectable errors are detected, according to some embodiments. In accordance with instruction set 312 of page retirement policy 124, control logic 210 instantiates patrol scrubber 122 to maintain fault map 510 that contains addresses of pages determined to contain either correctable or uncorrectable faults. In response to encountering a DUE in page 502 of DRAM-0 231, patrol scrubber 122 stores address 506 of page 502 containing the DUE in fault map 510.

[0043] The patrol scrubber 122 further references the fault map 510 to determine whether any neighboring pages, such as page 504, having a physical address within a predetermined distance of page 502 are recorded in the fault map 510. The predetermined distance, in some embodiments, is set by the page retirement policy 124. In some embodiments, the patrol scrubber 122 checks the neighbors of page 502 by scanning the fault map 510 for neighboring page addresses. In other embodiments, the fault map 510 is organized to track the start and end of a range of addresses of pages that are adjacent and found to have a DUE during the scrub interval. If the patrol scrubber 122 finds one or more neighboring pages of page 502 in the fault map 510, the patrol scrubber 122 swaps the neighboring pages with pages reserved to replace the retired page.

[0044] 6 is a flow diagram illustrating a method 600 for identifying faults in a subset of pages of a DRAM based on a fault map 510 that stores addresses of pages in which uncorrectable errors are detected, according to some embodiments. At block 602, a page retirement policy 124 sets the area that the patrol scrubber 122 will scrub and the scrubbing interval. At block 604, the patrol scrubber 122 detects a DUE in a page 502 of the DRAM. At block 606, the patrol scrubber 122 records the address of the page 502 in the fault map 510.

[0045] At block 608, the patrol scrubber 122 checks the fault map 510 for neighbors of page 502. Neighbors are identified based on having addresses within a predetermined distance of page 502, based on the page retirement policy 124. At block 610, if the patrol scrubber 122 finds a neighboring page of page 502 recorded in the fault map 510, method flow proceeds to block 612. At block 612, the patrol scrubber 122 swaps the neighboring page with the page reserved to replace the retired page. At block 610, if the patrol scrubber 122 does not find a neighboring page of page 502 in the fault map 510, method flow returns to block 604.

[0046] 7 is a block diagram 700 of control logic 210 that identifies a subset of DRAM pages to scrub based on a miss in TLB 120, according to some embodiments. As described above, TLB 120 stores a subset of virtual-to-physical address translations stored in page table 128. A miss in TLB 120 predicts a future miss in cache 112 that will require access to system memory 106. In accordance with instruction set 314, page retirement policy 124 instructs control logic 210 to identify a subset of DRAM pages to scrub based on a miss in TLB 120.

[0047] The control logic 210 identifies cache sets, called "miss sets," that are expected to have a large number of misses in the near future and that are expected to result in future accesses to the system memory 106. In some embodiments, as described below, any pages that have the same "set_index+offset" bit value are considered to be part of the "miss set."

[0048] Because a miss to the TLB 120 is more likely to result in a miss to the cache 112, in response to a miss to the TLB 120, the control logic 210 begins determining future misses to cache sets in the cache 112 before terminating the address translation process caused by the miss in the TLB 120. To predict future misses to cache sets, the control logic 210 first determines the index of a set that is likely to have a miss in the near future. In some embodiments, in response to a miss to the TLB 120, the control logic 210 uses a hash function to decode the page offset of the virtual address and then returns the index of the set. When the hash function generates a set number 706, the control logic 210 inserts the set number 706 into a TLB miss table 704. The TLB miss table 704 tracks sets referenced by TLB misses. The TLB miss table 704 includes the set number 706 inserted by the control logic 210 and a miss counter 708. In some embodiments, the miss counter 708 is a saturating counter that is incremented whenever there is a miss in a set. If the value of the miss counter 708 is greater than a predetermined set threshold 710, the set is considered a "miss set" 712. In some embodiments, the threshold 710 is defined by the programmer.

[0049] In some embodiments, the functions of the control logic are performed by IOMMU 118. When IOMMU 118 receives a TLB miss, IOMMU 118 uses logic to find the set number referenced by the TLB miss. IOMMU 118 then inserts the set number into a table of TLB miss sets 704 and updates a miss counter 708 corresponding to the set. The total number of bits in miss counter 708 is obtained in some embodiments using a log based on the associativity of cache 112.

[0050] Because only a limited number of sets are accessed in a particular time interval, TLB miss table 704 does not need to hold information for all sets; the size of TLB miss table 704 can be limited by storing only active sets in cache 112. In some embodiments, TLB miss table 704 stores only the total number of sets divided by a number N, and uses a direct-mapped cache structure to evict the least recently used lines from TLB miss table 704, where N is a constant defined by the programmer. For example, if cache 112 is a 2 MB cache with 16 associativity and a 64 B block size, there are 2048 sets (11 bits are used to hold the set index in the TLB miss table 704). Miss counter 708 has 4 bits. Assuming N=4 (so that 1 / 4 of the sets are stored in TLB miss table 704), the size of TLB miss table 704 is only 0.94 KB, which is approximately 0.046% of the size of cache 112.

[0051] 8-10 illustrate steps for extracting a set index from a virtual address. FIG. 8 is a diagram 800 illustrating the translation of a virtual address 802 to a physical address 812 to identify a subset of pages of DRAM for scrubbing based on a miss in the TLB 120, according to some embodiments. The control logic 210 determines a cache set number using a hash function before completing the address translation process. As shown in FIG. 8, during the translation of the virtual address 802 to a physical address 812, only the virtual page number 804 is translated to the physical page number 814; there is no translation of the virtual page offset 806 to the physical page offset 816 because the page offsets 806, 816 are the same for both the virtual address 802 and the physical address 812. The hash function uses only the page offset portion 806 of the virtual address 802 to determine the set number. Generally, the page offset is k bits, which is log(page size).

[0052] FIG. 9 is a diagram 900 illustrating the extraction of a set index 904 from a virtual page offset 806 to identify a subset of pages of DRAM to scrub based on a miss in the TLB 120, according to some embodiments. As shown in FIG. 9, in each cache physical address, a, b, and c bits are used for the byte offset 906, set index 904, and tag 902, respectively. If a+b is less than or equal to k, the control logic 210 can determine the exact set number. For smaller caches, or caches with higher associativity and fewer sets, the value of k is typically greater than or equal to a+b. However, if the value of a+b is greater than k, the hash function returns a group of set indices.

[0053] FIG. 10 is a diagram 1000 illustrating an example of a set index in which the number of page offset bits is less than the total number of bits required for the set index and offset to identify a subset of pages in DRAM to scrub based on a miss in the translation lookaside buffer, according to some embodiments. For example, as shown in FIG. 10, if (a+b)−k=2, there are two bits that have unknown values ​​until the virtual-to-physical address translation is complete. In this case, control logic 210 considers all four possibilities for the set number and inserts all four possibilities into TLB miss table 704 and increments their corresponding miss counters 708. Once a page in miss set 712 is identified, control logic 210 tasks patrol scrubber 122 with scrubbing the page in miss set 712 to determine whether the page contains any faults. If the patrol scrubber 122 encounters a DUE on any of the pages in the miss set 712, the patrol scrubber 122 retires all of the pages in the miss set 712.

[0054] 11 is a flow diagram illustrating a method 1100 for identifying a subset of pages of DRAM to scrub based on a miss in the TLB 120, according to some embodiments. At block 1102, the control logic 210 identifies a miss in the TLB 120. At block 1104, the control logic 210 decodes the page offset of the miss's virtual address 802 and returns the index of the set. At block 1106, the control logic 210 inserts the set number 706 into a table of TLB miss sets 704, and at block 1108, the control logic 210 increments the miss counter 708 of the miss set 712. At block 1110, the control logic 210 determines whether the miss counter 708 of the miss set 712 exceeds a threshold 710. If at block 1110 the control logic 210 determines that the miss counter 708 does not exceed the threshold 710, method flow returns to block 1102.

[0055] If, at block 1110, the control logic 210 determines that the miss counter 708 exceeds the threshold 710, method flow continues to block 1112. At block 1112, the patrol scrubber 122 scrubs the pages of the miss set 712. At block 1114, the patrol scrubber 122 determines whether there is a DUE in any of the pages of the miss set 712. If, at block 1114, the patrol scrubber 122 determines that one or more of the pages of the miss set 712 does not have a DUE, method flow returns to block 1102.

[0056] If, at block 1114, the patrol scrubber 122 determines that one or more of the pages in the miss set 712 has a DUE, then method flow proceeds to block 1116. At block 1116, the patrol scrubber 122 retires all of the pages in the miss set 712.

[0057] In some embodiments, the above-described apparatus and techniques are implemented in a system including one or more integrated circuit (IC) devices (also called integrated circuit packages or microchips), such as the processing systems described above with reference to FIGS. 1-11 . Electronic design automation (EDA) and computer-aided design (CAD) software tools can be used in the design and manufacture of these IC devices. These design tools are typically represented as one or more software programs. The one or more software programs include code executable by a computer system for operating the computer system to operate on code representing the circuits of one or more IC devices to perform at least a portion of a process for designing or adapting a manufacturing system for manufacturing the circuits. This code may include instructions, data, or a combination of instructions and data. The software instructions representing the design or manufacturing tools are typically stored on a computer-readable storage medium accessible to the computing system. Similarly, code representing one or more stages of the design or manufacture of the IC devices is stored on and accessed from the same or a different computer-readable storage medium.

[0058] A computer-readable storage medium includes any non-transitory storage medium or combination of non-transitory storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tape, magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical systems (MEMS)-based storage media. The computer-readable storage medium (e.g., system RAM or ROM) may be internal to the computing system, the computer-readable storage medium (e.g., a magnetic hard drive) may be permanently attached to the computing system, the computer-readable storage medium (e.g., an optical disk or Universal Serial Bus (USB)-based flash memory) may be removably attached to the computing system, or the computer-readable storage medium (e.g., network-accessible storage (NAS)) may be coupled to the computer system via a wired or wireless network.

[0059] In some embodiments, certain aspects of the techniques described above are implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied in a non-transitory computer-readable storage medium. The software may include instructions and specific data that, when executed by one or more processors, operate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as flash memory, a cache, a random access memory (RAM), or other non-volatile memory device(s). The executable instructions stored on the non-transitory computer-readable storage medium may be implemented as source code, assembly language code, object code, or other form of instructions that can be interpreted or otherwise executed by one or more processors.

[0060] In addition to the above, it should be noted that not all activities or elements described in the summary description are required, that some of the particular activities or devices may not be required, that one or more additional activities may be performed, and that one or more additional elements may be included. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will recognize that various modifications and variations can be made without departing from the scope of the invention as set forth in the claims. Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present invention.

[0061] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and features from which any benefit, advantage, or solution may arise or be manifested are not construed as critical, essential, or essential features of any or all claims. Moreover, the specific embodiments described above are illustrative only, since the disclosed invention may be modified and practiced in different, but similar manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the appended claims. It is therefore apparent that the specific embodiments described above may be altered or modified, and that all such variations are considered within the scope of the disclosed invention. Accordingly, the protection sought herein is set forth in the appended claims.

Claims

1. 1. A method comprising: responsive to detecting an uncorrectable error in a first page of a dynamic random access memory (DRAM), identifying a second page of the DRAM for potential retirement; method.

2. Identifying the second page includes: identifying a subset of pages of the DRAM that are a first physical distance from the first page, the second page being included in the subset of pages; checking for the presence of faults in the subset of pages; 10. The method of claim 1.

3. in response to detecting an uncorrectable error in the second page, retiring the second page. The method of claim 2.

4. Detecting uncorrectable errors in the first page is performed while checking for faults in the DRAM. The method of any one of claims 1 to 3.

5. recording the uncorrectable error detected in the first page in a fault map including a plurality of page addresses of faulty pages. The method of claim 4.

6. Identifying the second page includes: responsive to recording the uncorrectable error in the first page in the fault map, scanning the fault map for page addresses adjacent to the first page. The method of claim 5.

7. selecting a subset of page addresses for retirement based on the presence of an uncorrectable error in one or more pages of the subset, the subset including a page address of the second page. The method of claim 6.

8. Identifying the second page includes: identifying a subset of pages of the DRAM that are predicted to be accessed based on a miss in a translation lookaside buffer (TLB), the second page being included in the subset of pages; and retiring the subset of pages in response to detecting an uncorrectable error in the second page.

10. The method of claim 1.

9. 1. A method comprising: in response to detecting an uncorrectable error in a second page of a dynamic random access memory (DRAM), retiring the first page of the DRAM before accessing the first page of the DRAM. method.

10. identifying a subset of pages of the DRAM that are a first physical distance from the second page, the first page being included in the subset of pages; checking for the presence of faults in the subset of pages; 10. The method of claim 9.

11. in response to detecting an uncorrectable error in the second page, retiring the first page. The method of claim 10.

12. detecting the uncorrectable error in the first page while checking for faults in the DRAM. The method of any one of claims 9 to 11.

13. recording the uncorrectable error detected in the first page in a fault map including a plurality of page addresses of faulty pages.

13. The method of claim 12.

14. responsive to recording the uncorrectable error in the second page in the fault map, scanning the fault map for page addresses adjacent to the second page.

14. The method of claim 13.

15. selecting a subset of page addresses for retirement based on the presence of an uncorrectable error in one or more pages of the subset, the subset including a page address of the first page.

15. The method of claim 14.

16. identifying a subset of pages of the DRAM that are predicted to be accessed based on a miss in a translation lookaside buffer (TLB), the first page being included in the subset of pages; and retiring the subset of pages in response to detecting an uncorrectable error in the first page.

10. The method of claim 9.

17. A device, Dynamic random access memory (DRAM); a control logic circuit; The control logic circuit configured to identify a second page of the DRAM for potential retirement in response to detecting an uncorrectable error in a first page of the DRAM. device.

18. The control logic circuit identifying a subset of pages of the DRAM that are a first physical distance from the first page, the second page being included in the subset of pages; checking for the presence of faults in the subset of pages; configured to:

18. The device of claim 17.

19. The control logic circuit recording the uncorrectable error detected in the first page in a fault map including a plurality of page addresses of faulty pages; responsive to recording the uncorrectable error in the second page in the fault map, scanning the fault map for page addresses adjacent to the second page; selecting a subset of page addresses for retirement based on the presence of an uncorrectable error in one or more pages of the subset, the subset including a page address of the first page; configured to:

18. The device of claim 17.

20. The control logic circuit identifying a subset of pages of the DRAM that are predicted to be accessed based on a miss in a translation lookaside buffer (TLB), the second page being included in the subset of pages; retiring the subset of pages in response to detecting an uncorrectable error in the second page; configured to:

18. The device of claim 17.

Images