Low cost high performance LRAID

Abstract

The invention relates to a low cost high performance LRAID. A memory access method with improved bandwidth efficiency for reliability, availability, and maintainability RAS is described. The memory access method includes: in response to a memory access request, obtaining sub-stripes from a data stripe stored on a plurality of memory media components arranged in a redundant array of independent data RAID using a first access granularity; and detecting an error in the sub-stripe of the data. The method further includes, in response to the detecting the error, obtaining the stripe of the data from the plurality of memory media components using a second access granularity, where the second access granularity is greater than the first access granularity. A corresponding apparatus is also described.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 512,571, filed on July 7, 2023, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to locked redundant array of independent devices (LRAID) type storage media. In particular, the present disclosure relates to LRAID with reliability, availability and serviceability (RAS) features for storage media. Background Art

[0004] Memory media (also referred to as "memory media devices") are widely used to store information in various electronic devices, such as computers, user devices, wireless communication devices, cameras, digital displays, and the like. Information is stored by programming memory cells within the memory media device to various states. For example, a binary memory cell can be programmed to one of two supported states, typically corresponding to a logical 1 or a logical 0. In some examples, a single memory cell can support more than two possible states, either of which can be stored by the memory cell. To access information stored by a memory media device, a component can read or sense the state of one or more memory cells within the memory media device. To store information, a component can write or program one or more memory cells within the memory media device to a corresponding state.

[0005] There are various types of memory media devices, including magnetic hard disks, random access memory (RAM), read-only memory (ROM), dynamic RAM (DRAM), synchronous dynamic DRAM (SDRAM), static RAM (SDRAM), flash memory, and others. Memory media devices can be volatile or non-volatile. Volatile memory cells (e.g., DRAM cells) can lose their programmed state over time unless they are periodically refreshed by an external power source. DRAM is organized into an array of memory cells, where each cell stores a programmed value. SRAM memory can maintain its programmed state for the duration that the system is powered on. Non-volatile memory cells (e.g., NAND memory cells) can maintain their programmed state for an extended period of time even in the absence of external power.

[0006] RAS features are an important consideration in many digital memory storage environments. The explosive growth of Internet content and the resulting data storage and computing needs have led to the deployment of heterogeneous and sometimes complex data storage solutions in large-scale data centers, organizations, home computing environments, and even mobile computing environments. The increasing scale, intensity of use, importance of applications, etc. of such deployments have led to increasing attention to their RAS capabilities.

[0007] In some modern deployments, as the total number of components and bits stored or transmitted continues to increase, even traditionally uncommon errors are starting to become visible. Since the number of repair events requiring hardware replacement should be kept to an absolute minimum, there is a need to clearly and unambiguously identify failed components and minimize the time spent on repair.

[0008] Reliability is defined as the ability to provide correct service. One way to design a reliable system is to be able to detect and correct faults that would otherwise cause errors. An example of this design is the use of error correction code (ECC) to detect faults at specific locations in memory and correct the data stored in that location before it is consumed by the processor. Reliability is often expressed in terms of mean time between failures (MTBF), mean time to failure (MTTF), or failure in time (FIT).

[0009] Availability is defined as the ability to be ready to provide correct service, possibly at a reduced capacity or performance. One way to achieve availability is to recover from errors so that service is not interrupted due to errors. Using the memory example above, if an error is detected but cannot be corrected by the ECC, then we can take the physical page containing a specific memory address in the operating system offline (retire) so that no data is allocated to that location in the future. If the recovery is unsuccessful and the machine crashes and resets itself, then this will result in a period of downtime, which means that the system is unavailable during this downtime. One way to quantify availability is in terms of "number of 9s." For example, 5 9s of availability means that the system is up and running 99.999% of the time, which is approximately 5.25 minutes of downtime in one year of system operation.

[0010] Serviceability is defined as the ability to diagnose and repair faults that may have caused (or may have caused) errors. For example, memory repair techniques that allow a memory location with an uncorrectable permanent fault to be physically repaired by replacing it with a spare would provide serviceability. Serviceability can be "online," where the system continues to be available, or "offline," where the system is unavailable while maintenance is in progress.

[0011] The specific mechanisms used to achieve RAS can be a mix of process technology, architecture, firmware, system software, libraries, and potentially application software. Designing RAS for multiple use cases is done in accordance with standards, such as Compute Express Link TM (CXL TM ) ecosystem, standardization of certain canonical RAS features, high visibility into errors and other RAS-related events within a set of components interconnected by the CXL fabric, and the ability to log these events in a standardized manner and pass them to the host to enable RAS actions within platform-level and datacenter-level operational flows.

[0012] CXL specifies standards for System on Chip (SoC), memory, storage, networking, etc. CXL enables system vendors to populate memory media devices with different internal architectures and media technologies in the same platform. We can also populate different coherent accelerators, each with potentially different types of local memory and from different vendors. The CXL standard defines multiple RAS features, enabling all members of this rich heterogeneous ecosystem to innovate in their domains and still interoperate, ultimately leading to RAS design points that meet current and emerging end-user needs.

[0013] Memory media failures can be classified into two categories - random (e.g., single bit) errors and peripheral errors. Random errors can be soft errors or hard errors and can be in the memory array or on the data link. Peripheral errors are caused by such aspects as neutron strikes, defects in latches, metal contamination, and imbalances in sense amplifiers. As described above, system resilience is usually determined according to RAS criteria. According to RAS criteria, reliability for random and peripheral errors is provided by 3 mechanisms: ECC - acts on a payload consisting of bits from a single memory media die; Low Power Chip Kill (LPCK) - corrects entire die failures and provides countermeasures for peripheral errors in the memory media; and Cyclic Redundancy Check (CRC) - protects the CXL link and prevents errors from reaching the host.

[0014] Storage media devices are arranged in a scheme such as a redundant array of independent data (RAID) to disperse or stripe data across several storage media devices to improve access speed and reduce the possibility of errors. LRAID systems access storage media devices arranged in a RAID configuration at a specifically configured access granularity. Although LRAID provides advantages in speed and reliability, some configurations require accessing substantially large amounts of data in storage media devices to obtain the much smaller amount of data required for a read or write request, resulting in heavy utilization of the interfaces of the storage media devices.

[0015] In view of the importance of the above-described RAS capabilities, there exists a need for improved solutions for providing such capabilities in LRAID storage media device arrangements. Summary of the invention

[0016] One aspect of the present disclosure relates to a memory access method, comprising: in response to a memory access request, obtaining a sub-stripe from a data stripe stored on multiple storage media components arranged in an independent data redundant array (RAID) using a first access granularity; detecting an error in the sub-stripe of the data; and in response to detecting the error, obtaining the stripe of the data from the multiple storage media components using a second access granularity, wherein the second access granularity is greater than the first access granularity.

[0017] In some embodiments, the stripe of data includes a plurality of blocks of the data, and wherein each block of the plurality of blocks and parity information for the stripe are stored on a respective storage media component of the plurality of storage media components.

[0018] In some embodiments, obtaining a sub-stripe using a first access granularity and obtaining a stripe using a second access granularity is consistent with locked RAID (LRAID) access.

[0019] In some embodiments, the size of the data block in the memory access request is 64 bytes, the size of the first access granularity is 256 bytes, and the size of the second access granularity is 512 bytes.

[0020] In some embodiments, a memory access method is performed by a memory controller in a device configured to operate as a CXL Type 3 device.

[0021] In some embodiments, the memory media component is a dynamic random access memory (DRAM) media.

[0022] Another aspect of the disclosure relates to a device, comprising a host interface, the host interface including: a plurality of input / output (I / O) paths configured to couple to a host device; a physical (PHY) layer interface configured to couple to a plurality of memory media components; and a control circuit system configured to perform operations. This includes: in response to a memory access request received through the host interface, obtaining a sub-stripe from a stripe of data stored on the plurality of memory media components arranged in an independent data redundant array (RAID) using a first access granularity and the PHY layer interface; detecting an error in the sub-stripe of the data; and in response to detecting the error, obtaining the stripe of the data from the plurality of memory media components using a second access granularity and the PHY layer interface, wherein the second access granularity is greater than the first access granularity.

[0023] In some embodiments, the control circuitry is further configured to identify a location of an error in a sub-stripe using a stripe obtained using the second access granularity and parity information for the stripe.

[0024] In some embodiments, the central controller is further configured to send a Compute Express Link TM (CXL TM ) message to notify the sender of the memory access request about the detected error.

[0025] In some embodiments, CXL messages conform to the reliability, availability, and serviceability (RAS) capabilities of the CXL protocol.

[0026] In some embodiments, the device further comprises a cache, wherein the central controller is further configured to store the sub-stripe in the cache, and wherein the first access granularity or the second access granularity corresponds to a size of a cache line of the cache.

[0027] In some embodiments, the control circuitry is further configured to access a stripe, each of a plurality of sub-stripes of the stripe, and a parity of the stripe, respectively, in each storage media component of the plurality of storage media components according to locked RAID (LRAID) access. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The illustrative embodiments may take the form of various components and arrangements of components. The illustrative embodiments are shown in the accompanying drawings, and the same reference numerals may indicate corresponding or similar parts in the various drawings throughout the drawings. The drawings are for the purpose of illustrating the embodiments only and should not be construed as limiting the present disclosure. Given the following illustrative description of the drawings, the novel aspects of the present disclosure should be apparent to one of ordinary skill in the art.

[0029] Figure 1 Example functional block diagram illustrating the calculation.

[0030] Figure 2 The following diagrams illustrate some embodiments of the present disclosure. Figure 1 Detection of data errors in a RAID array is demonstrated.

[0031] Figure 3 The following diagrams illustrate the embodiment of the present disclosure in the example Figure 1 An example of a LRAID configuration of a storage media device showing a first memory access for data using a first access granularity and a second memory access for error recovery using a second access granularity.

[0032] Figure 4 The following diagrams illustrate the use of some embodiments of the present disclosure when used to read Figure 3 A scenario in which the first memory access to the data referenced in does not have a data error.

[0033] Figure 5 The following diagrams illustrate the use of some embodiments of the present disclosure when used to read Figure 3 A scenario in which the first memory access to the data referenced in has a data error.

[0034] Figure 6 The following diagrams illustrate some embodiments of the present disclosure when used in, for example, Figure 1 A scenario is shown when the first memory access to write data in an LRAID configuration has no data errors.

[0035] Figure 7 A scenario when a first memory access for writing data has a data error according to some embodiments of the present disclosure is graphically illustrated.

[0036] Figure 8 Storage media devices in an LRAID arrangement for LRAID16+1 with 2 LRAID8+1 sub-stripes according to some embodiments of the present disclosure are described.

[0037] Fig. 9 Storage media devices in an LRAID arrangement for LRAID32+1 with 4 LRAID8+1 sub-stripes according to some embodiments of the present disclosure are described.

[0038] Fig.10 The following describes some embodiments of the present disclosure executed at a target CXL controller to serve, for example, Figure 1 Flowchart of the process of a read request in a system in FIG.

[0039] Fig.11 The following describes some embodiments of the present disclosure executed at a target CXL controller to serve, for example, Figure 1 Flowchart of the process of a write request in a system in FIG. DETAILED DESCRIPTION

[0040] Memory solutions in which multiple memory media devices (e.g., DRAM or other emerging memory dies) are incorporated into or associated with one memory device (e.g., such as a memory controller) are growing in size and importance. With this growth, it becomes increasingly important to effectively and efficiently implement RAS capabilities in such memory solutions.

[0041] Embodiments of the present disclosure relate to accessing multiple storage media devices arranged in a RAID organization.Example embodiments provide efficient error detection and correction, and thus provide RAS with improved bandwidth utilization.Some embodiments provide CXL compatible memory devices that utilize the RAS capabilities of CXL.

[0042] Figure 1 An example functional block diagram of a computing system 100 is illustrated according to an example embodiment of the present disclosure, the computing system 100 including an interface host CXL initiator controller 110, a target CXL controller 122, and a CXL communication interface 106 connecting the host and target CXL devices, which are configured to improve bandwidth utilization of data requests in a RAS-enabled LRAID storage arrangement. According to an embodiment, the CXL communication interface 106 is a CXL interface configured to connect the CXL initiator (also referred to as a CXL host or CXL master) controller 110 and the target (also referred to as a CXL slave) CXL controller 122 according to, for example, the CXL 2.0 and / or CXL 3.0 protocol specifications. However, example embodiments are not limited to the above versions of CXL or the CXL protocol.

[0043] The host 102 may be a computer or the like having one or more processors 103 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit system (ASIC), a field programmable gate array (FPGA), or another type of processor). The host 102 may further include a host component device 108, such as a cache memory, for example.

[0044] The host 102 may be configured to access a device, such as, for example, a memory device 104, such as, for example, a memory controller or a system on a chip (SoC) including a memory controller. The memory device 104 may include one or more memory media devices 118, such as DRAM modules, which may be LPDDR, DDR, or other types of DRAM. In some embodiments, the memory device 104 may be configured as a "main memory" or some other memory for the host 102. Example embodiments are not limited to DRAM, and may, for example, include a memory device 104 having one or more memory media modules or a combination of memory types.

[0045] Host 102 and memory device 104 communicate via CXL communication interface 106. In some embodiments, CXL communication interface 106 includes two unidirectional links, one for host 102 to transmit messages to memory device 104, and the other for memory device 104 to transmit messages to host 102.

[0046] As mentioned above, in some embodiments, the memory device 104 is a CXL-compatible memory system (e.g., the memory system may include a PCIe / CXL interface). For example, the front end portion 112 of the memory device 104 includes a PCI / PCIe interface 120 and a target CXL controller 122. The PCI / PCIe interface 120 couples the memory device 104 to the host 102 through one or more input / output (I / O) pathways of the CXL communication interface 106. For example, communication through the I / O pathways may be according to a protocol such as Peripheral Component Interconnect Express (PCIe). In some embodiments, multiple I / O pathways may be configured as a single port. Example embodiments may not be limited by the number of I / O pathways, whether the I / O pathways belong to a single port, or the communication protocol used to communicate with the host.

[0047] CXL is a dynamic multi-protocol technology designed to support accelerators and memory devices. In the CXL protocol, the Compute Fast Link Layer (Link Layer) is responsible for reliably transmitting transaction layer packets (TLPs) across the Flex Bus link. CXL provides a rich set of protocols that include I / O semantics similar to PCIe (i.e., CXL.io), cache protocol semantics (i.e., CXL.cache), and memory access semantics (i.e., CXL.mem) on discrete or packaged links. CXL.io is required for discovery and enumeration, error reporting, and host physical address (HPA) lookups. The CXL.mem and CXL.cache protocols can optionally be implemented by a specific accelerator or memory device use model.

[0048] CXL can be viewed as a high-speed CPU-to-device and CPU-to-memory interconnect designed to accelerate next-generation data center performance. CXL technology maintains memory coherency between the CPU memory space and the memory on the attached device, which allows resource sharing to achieve higher performance, reduce software stack complexity, and reduce overall system cost. CXL is designed as an industry open standard interface for high-speed communication as accelerators are increasingly used to supplement CPUs to support emerging applications such as artificial intelligence and machine learning.

[0049] CXL technology is built on the PCIe infrastructure, utilizing the PCIe physical and electrical interfaces to provide high-level protocols in areas such as I / O protocols, memory protocols (e.g., initially allowing hosts and accelerators to share memory), and coherent interfaces. When the memory device 104 is CXL-compatible, the interface management circuitry (e.g., PCI / PCIe interface 120) associated with the target CXL controller 122 (including data link and transaction control) uses the CXL protocol to manage the CXL communication interface 106, which may include a PCIe PHY interface.

[0050] Each message transmitted between the host 102 and the memory device 104 through the CXL communication interface 106 can be broken down into smaller data structure blocks called packets. In the examples described in the present disclosure, each packet can be broken down into message flow control units or "flits". The flow control mechanism can decide which message to flow and which message to retain. A "flit" (flow control unit) is a link-level data structure that forms a network packet or flow. A flit includes multiple slots, and each slot contains header information or data for a message.

[0051] A flits may contain one or more messages. In some embodiments, a CXL read request may occupy one (1) slot in a flits, and a CXL write request may occupy five (5) slots. For example, for a write request, one (1) slot may be occupied by the header of the write request, and four (4) slots may be occupied by the data to be written. In the same example, in the upstream flits (i.e., the flits from the memory device 104 to the host 102), since the write response does not contain data, the write response occupies one (1) slot, and the read response occupies five (5) slots, with one (1) slot being used for the read response header and four (4) slots being used for the returned data. It should be noted that CXL specifies rules for how to pack the flits slots.

[0052] According to the CXL protocol, flits can be accepted or rejected on the receiver side based on the flow control protocol and the size of the receiver buffer. The link-level flow control mechanism allows the receiver to send a continuous stream of signals to the transmitter to control whether it should keep sending flits or stop sending flits. When transmitting a packet over a link, it is usually necessary to split the packet into multiple flits before starting the transmission.

[0053] For CXL.cache and CXL.mem, a layout is specified, such as a 528-bit flits layout, for example. Flit packing rules for selecting requests from an internal queue to fill available slots in a flits (e.g., 3 slots per flits in one example CXL implementation, 16 slots per flits in another example CXL implementation) are also specified. Other features described for CXL.cache and CXL.mem include a retry mechanism, link layer control flits, CRC calculations, and viruses and poisons. A major advantage of CXL is that it provides a low latency, high bandwidth path for accelerators to access systems and for systems to access memory attached to CXL devices.

[0054] The flexible bus provides a point-to-point interconnect that can carry either native PCIe protocol or dynamic multiprotocol CXL to provide I / O, cache, and memory protocols over the PCIe electrical device. Key link attributes include support for the following features: Native PCIe mode, full feature support as defined in the PCIe specification; CXL mode, as defined in this specification; configuration of PCIe and CXL protocol modes; signaling rate of 32GT / s; downgraded rates of 16GT / s or 8GT / s in CXL mode; link width support of x16, x8, x4, x2 (downgraded mode), and x1 (downgraded mode).

[0055] When operating as a CXL interface based on PCIE 5.0 / 6.0, CXL communication interface 106 includes a pair of unidirectional links, each of which contributes to system bandwidth. CXL 2.0 provides an interface utilizing PCIE 5.0 (32GT / s), and CXL 3.0 utilizes PCIE 6.0 (64GT / s) in each direction. According to some embodiments in which CXL communication interface 106 operates in accordance with CXL, memory device 104 is a CXL type 3 device. This means that the interface operates in a host-target (also known as master-slave) mode in which a CXL initiator controller 110 in host 102 operates as a host and a target CXL controller 122 in memory device 104 operates as a target. In embodiments in which the memory device is a CXL type 3 device, all requests (e.g., read requests and write requests) are initiated by host 102, and memory device 104 responds to such requests. However, example embodiments are not limited to a host-target configuration.

[0056] Multiple requests and data are transmitted over the CXL communication interface 106 in flits, which in some embodiments are fixed width flits. The host (e.g., CXL initiator) is responsible for generating requests to the memory device (e.g., CXL target). The memory device has a CXL controller that unpacks incoming flits requests from the host and forwards the requests to the central controller and backend memory media (e.g., DRAM). Each flit in CXL 2.0 is 528 bits (4 slots, 16B per slot + 2 bytes for CRC = 66B or 528 bits).

[0057] Each flits in CXL 3.0 is 256B (16 slots, 16B per slot, with 238B of each flits used for TLP payload and 18B for CRC and forward error correction (FEC)). In some embodiments, in CXL 3.0, 15 of the 16 slots can be used for multiple requests and data. In an example embodiment, the CXL initiator controller 110 and the target CXL controller 122 are configured to implement the functionality required for CXL. For example, the CXL initiator controller 110 and the target CXL controller 122 operate to encapsulate messages or TLPs into flits for transmission on the CXL communication interface 106, and to unpack flits received on the CXL communication interface 106 to obtain the transmitted message.

[0058] According to some embodiments, the memory media devices 118a, 118b, ..., 118p (collectively or sometimes individually referred to as 118, p is any number greater than or equal to 2) include one or more DRAM devices, which may be of a type not limited to, for example, DDR, LPDDR, etc., as described above. In some embodiments, the main memory of the computer system 100 is stored in DRAM cells with a high storage density. Over time, DRAM cells lose their state. That is, DRAM cells must be refreshed periodically, and are therefore called dynamic. DRAM is organized according to a hierarchy of storage organization, which includes dual in-line memory modules (DIMMs), ranks, memory banks, and arrays.

[0059] A DIMM contains multiple DRAM chips, and the multiple chips in a DIMM are organized into one or more "ranks". Each chip is formed of multiple "banks". A bank is formed of one or more "rows" of memory cell arrays. All banks within the rank share all address and control pins. All banks are independent, but in some embodiments, only one bank in a rank can be accessed at a time. Due to electrical limitations, only a few DIMMs can be attached to a bus. Ranks help increase the capacity on a DIMM.

[0060] In an example embodiment, multiple DRAM chips are used for each access to improve data transfer bandwidth. Multiple banks are provided so that the computing system can process different requests simultaneously. To maximize density, the arrays within the banks become large, the rows are wide, and the row buffers are wide (8KB read for 64B requests). Each array provides a single bit to the output pin within a cycle (for high density and because there are few pins). DRAM chips are often described as xN, where N refers to the number of output pins; a row can be composed of 8x8 DRAM chips (for example, the data bus is 64 bits). Banks and rows provide memory parallelism, and the memory device 104 can schedule memory accesses to maximize row buffer hit rates and bank / row parallelism.

[0061] In some embodiments, the memory media device 118 is a low power double data rate (LPDDR) LP5 or other similar memory interface. However, the embodiments are not limited thereto, and the memory media device 118 may include one or more memory media of any memory media type, such as but not limited to DRAM.

[0062] The memory device 104 includes a memory media controller and other circuitry to control the performance of memory operations in response to requests or commands received from the host 102. The memory operations may be memory access operations to read data from or write data to the memory media device 118.

[0063] The back end portion 116 of the memory device 104 may include a plurality of media controllers 134 (134a ... 134n, n being any number 2 or greater) and a plurality of PHY interfaces 136 (136a ... 136n), such that each of the media controllers 134 controls a group of one or more memory media devices 118 via a respective one of the PHY interfaces 136. Each of the plurality of media controllers 134 may receive the same commands (e.g., commands to read or write to the memory media device block 118), and address and drive a plurality of channels connected to the memory media substantially simultaneously.

[0064] By using the same commands and addresses for multiple media controllers, each of the multiple media controllers 134 can perform the same memory operations on the same multiple memory units using multiple channels. Each of the media controllers 134 and / or each group of one or more of the multiple channels can be associated with a corresponding RAID array.

[0065] As used herein, the term "substantially" intends that a characteristic need not be absolute but close enough to achieve the advantage of the characteristic. For example, "substantially simultaneously" is not limited to operations performed simultaneously and may include timing that is intended to be simultaneous, but due to manufacturing limitations, may not be exactly simultaneous.

[0066] For example, due to the read / write latencies that various interfaces (e.g., LPDDR5 and PCIe) may exhibit, media controllers that are utilized substantially simultaneously may not start or end at exactly the same time. For example, multiple media controllers may be utilized such that they write data to a memory device simultaneously regardless of whether one of the media controllers starts or ends before another.

[0067] The back end portion 116 is configured to couple to the storage media device 118 via (e.g., through) a plurality of channels 140, which may be used to read / write data from / to the storage media device 118, transmit commands to the storage media device 118, receive status and statistics from the storage media device 118, etc. The back end portion includes a media controller portion including one or more media controllers 134 and a physical (PHY) layer portion including a plurality of PHY interfaces 136 that connect the media controllers to the storage media device 118. In some embodiments, the back end portion is configured to couple the PHY interfaces to a plurality of memory banks of the storage media device 118.

[0068] The memory banks may be connected to the media controllers 134 via a plurality of channels 140. The respective media controllers 134 and corresponding PHY interfaces may drive the channels to the memory banks. In some embodiments, each media controller 134 may execute commands independently of the other media controllers 134. Thus, data may be transferred from one PHY interface to the memory media device 118 via channels independent of the other PHY interfaces and channels.

[0069] Each PHY interface 136 may operate according to a PHY layer that couples the memory device 104 to one or more memory banks in the memory media device 118. As used herein, the term "PHY layer" generally refers to a physical layer in the open system interconnection (OSI) model of computing systems. The PHY layer may be the first (e.g., lowest) layer of the OSI model and may be used to transmit data on a physical data transmission medium. In some embodiments, the physical data transmission medium may be a plurality of channels.

[0070] As used herein, the term "memory rank" generally refers to a plurality of memory chips (e.g., DRAM memory chips) that can be accessed simultaneously. In some embodiments, the memory rank may be sixty-four (64) bits wide, and each memory rank may have eight (8) pages. In some embodiments, the page size of a first type of memory media device may be larger than the page size of a second type of memory media device. However, example embodiments are not limited to a particular width or page size of a memory rank.

[0071] Each media controller 134 may include channel control circuitry and multiple bank control circuitry, wherein a respective one of the multiple bank control circuitry is configured to access a respective one of the multiple banks of memory on the memory media device 118 accessed by the respective media controller 134 .

[0072] Ranks, channels, and banks can be viewed as hardware-dependent logical groupings of storage locations in a media device. In some embodiments, the mapping of the rank, channel, and bank logical groupings to physical storage locations or rows in the memory media device can be preconfigured or configured by a memory controller in communication with the memory media device 118.

[0073] In some embodiments, each media controller in the plurality of media controllers 134 may correspond to a respective RAID array of the RAID circuitry 132. Each media controller in the plurality of media controllers 134 may also correspond to one of the plurality of PHY interfaces 136. Each PHY in the plurality of PHY interfaces 136 may be coupled to a memory media device 118 (e.g., a DRAM). In some embodiments, each media controller in the plurality of media controllers 134 may execute commands independently of the other media controllers. Thus, data may be transmitted from the PHY layer to the memory media device 118 via one of the plurality of channels 140 independently of the other PHYs in the plurality of PHY interfaces 136.

[0074] Memory device 104 includes a central controller portion 114 located between the circuitry of front end portion 112 and the circuitry of back end portion 116. Central controller portion 114 may include a media management layer / interconnect (MML layer) 124, cache 126, RAS circuitry 128, ECC circuitry 130, and RAID circuitry 132.

[0075] The MML layer 124 may include a crossbar interconnect that can direct corresponding memory access requests to corresponding memory media devices 118. The cache 126 temporarily stores data associated with the execution of memory operations to make memory access more efficient. In some embodiments, in response to receiving a request from the host 102, data read from or written to the memory media device 118 may be stored in a cache line of the cache 126 on the memory device 104. The data in the cache 126 may be written to the memory media device 118 according to various cache policies. In some embodiments, the ECC circuit system 130 on the memory device is configured to provide error correction for data read from and / or written to the memory media device 118.

[0076] Central controller portion 114 also includes RAID circuitry 132. RAID circuitry 132 may include multiple RAID components to store data on storage media devices 118. As used herein, the term RAID component generally refers to a data storage virtualization technology that combines multiple storage media components ("media components"), such as physical memory chips, memory banks, or other storage units, into one or more logical units for the purpose of data redundancy, performance improvement, or both.

[0077] In some embodiments, central controller portion 114 and / or RAID circuitry 132 may include a finite state machine (FSM) and multiple RAID components, wherein the FSM is operable to increase the reliability of data and correct errors in the data. Although illustrated as a single component, RAID circuitry 132 may include multiple RAID components, wherein each RAID component controls a group of storage media devices 118 in a RAID array according to the FSM.

[0078] In some embodiments, each RAID array can be coupled to the circuitry from ECC circuitry 130 in a manner independent of the ECC / CRC processing of other RAID arrays in RAID circuitry 132. The ECC circuitry may have a detection capability that triggers the RAID mechanism each time an error is detected in the user data pattern. In addition, the ECC circuitry may also have additional correction capabilities to correct a subset of errors detected in the decoding phase. In some embodiments, the ECC is reduced to a CRC code that only allows detection of errors and triggers the RAID recovery process. In some embodiments, each RAID array controlled by RAID circuitry 132 may correspond to a media controller group in media controller 134.

[0079] In an embodiment, the RAID circuit system 132 is configured to provide multiple RAID arrays, and each RAID array controls access to a group of 17 storage media devices 118. That is, the RAID array is striped across 16 storage media devices 118, wherein another of the storage media devices is used to store parity data. In this example arrangement, the group of 17 storage media devices 118 can be accessed via one of the channels 140. One of the media controllers 134 and one of the PHY interfaces 136 provide connections between the RAID array and the corresponding group of 17 storage media devices 118. This example arrangement allows a separate RAID array and a separate media controller 134 of the RAID circuit system 132 to be dedicated to each of the channels 140. The RAID state machine can implement the functionality of the RAID circuit system 132.

[0080] By dedicating a separate RAID array and a separate media controller 134 from the RAID circuit system 132 to each channel 140, each channel 140 can be driven individually and receive commands and addresses different from other channels 140. In some embodiments, each media controller 134 executes commands independently of other media controllers 134. Compared with the LPCK architecture, this RAID architecture can provide greater flexibility to the memory system in terms of how much data is written to the memory media device 118 and when the data is written to the memory media device 118. In some embodiments, the RAID array can be striped across multiple channels 140. If the RAID array is striped across multiple channels 140, the RAID state machine can be shared across multiple channels 140. This allows the RAID array to drive multiple channels 140 substantially simultaneously.

[0081] The central controller portion 114 may also include a security component configured to encrypt data before storing the data in the storage media device 118 and decrypt the data after reading the data in the storage media device 118. In some embodiments, an encryption protocol may be used to encrypt / decrypt the data, such as Advanced Encryption Standard (AES) encryption.

[0082] A management unit (not shown separately) located in the memory device 104 may be configured to control the operation of the memory device 104. The management unit may recognize commands from the host 102 and manage the one or more storage media devices 118 accordingly. In some embodiments, the management unit includes an I / O bus for managing out-of-band data, a management unit controller for executing firmware (whose functionality includes but is not limited to monitoring and configuring the characteristics of the memory device 104), and a management unit memory for storing data associated with the functionality of the memory device 104. The management unit controller may also execute instructions associated with initializing and configuring the characteristics of the memory device 104.

[0083] The management unit may couple the memory device 104 to an external circuit system or an external device, such as the host 102, which may generate requests to read data from the memory medium and / or write data to the memory medium, by initializing and / or configuring the memory device 104 and / or the memory media device 118 accordingly. The management unit is configured to recognize commands received from the host 102 and execute instructions to apply a specific operation code associated with the received host command to each of the plurality of channels coupled to the memory media device 118.

[0084] RAS circuitry 128 provides RAS functionality to recover from errors and / or notify host 102 about errors detected, for example, by ECC circuitry 130 in a RAID array of RAID circuitry 132. For example, when a parity error is detected in data obtained from one of the RAID arrays, RAS circuitry 128 may operate with ECC circuitry 130 and RAID circuitry 132 to recover from the error. In another example, when an unrecoverable error is detected in such obtained data, RAS circuitry 128 may notify host 102 of the unrecoverable error (e.g., using a CXL poison message).

[0085] Figure 2 The operation of locking RAID according to some embodiments of the present disclosure is illustrated by diagrams and examples. Figure 1 Theory of detection of data errors in a locked RAID array is presented.

[0086] The illustrated scenario is for an example LRAID4+1 configuration. In this configuration, memory accesses (e.g., read requests and write requests) are made to a RAID stripe (sometimes simply referred to as a "stripe") that includes data (e.g., user data) stored in four media components and parity data residing on another media component. The media component may be a corresponding one of the memory media devices 118a...118p (collectively and sometimes individually referred to as 118) or a corresponding memory bank in a plurality of memory banks of the memory media device 118. Each block A, B, C, D comes from a corresponding media component and may be a user data block of a fixed size (e.g., 32 bytes, 64 bytes, etc.). The parity block P may come from another media device in the same RAID stripe.

[0087] In LRAID, access to media components is locked to a specific access granularity. In this example, the access granularity can be considered as 32 bytes x 4 media components, plus the byte size of the parity data P.

[0088] Defines the read size and burst length for memory accesses. Figure 2 In the illustrated embodiment of , a burst size of 32 bytes and a read size of 1 byte may be assumed. Thus, each read operation reads 32x1=32 bytes of data from the memory components to the cache or other portion in the memory controller. Since in a RAID array, all memory components of a stripe containing user data are retrieved in a read access, 32 bytes will be read from each of the 4 data blocks A, B, C, and D in the LRAID4+1 array scenario.

[0089] The parity data for a stripe includes parity information for all four media components that are part of the stripe. In an embodiment, the parity data may be calculated as a bitwise exclusive OR (XOR) of all user data. More specifically, for example, a first bit of the parity data corresponds to an XOR sum of a first bit of each of four data blocks (e.g., user data residing on corresponding media components of the stripe), a second bit of the parity data corresponds to an XOR sum of a second bit of each of the four data blocks, and so on.

[0090] In a configuration where data accessed from a media component is written to a cache, the RAID stripe may be sized according to the size of a cache line. For example, when a cache has a 256-byte cache line, the RAID stripe may be 256 bytes, 512 bytes, etc. When a data element (e.g., a particular block of data) needs to be accessed in the RAID array, the entire stripe containing the data element is loaded into the cache (or elsewhere in the memory controller).

[0091] exist Figure 2 , on the left, data obtained from a RAID stripe of 4 media components storing data (e.g., user data) is shown, and another media component storing parity data is shown. Each block A, B, C, D may come from a corresponding media component (e.g., a corresponding one of the storage media devices 118a...118p, collectively and individually referred to as 118), and may be a data block of a fixed size (e.g., 32 bytes, 64 bytes, etc.). The parity block P may come from another media component in the same RAID stripe. In some embodiments, the parity data may include a fixed number of bits for each data block in the stripe. For example, in an embodiment, each user data block and parity data block in the same stripe have the same fixed size, and the parity is calculated as the bitwise XOR of all user data blocks in the stripe.

[0092] Data transfers between a media component (e.g., storage media device 118) and a cache (e.g., cache 126) may include error checking information, such as, for example, a CRC check. CRC checking is a known error detection and correction technique used in data transfers. In some embodiments, the CRC information for a stripe is stored in association with the user data for the stripe in a reserved area in the media and / or cache line. Figure 2 In the illustrated scenario, the CRC check indicates that the obtained stripe has errors. However, the CRC check alone cannot identify the erroneous blocks.

[0093] Example embodiments exploit this property: when the parity data for a stripe is the bitwise XOR of all user data blocks in the stripe, the user data for a particular user data block in the stripe is generated by the sum of all user data blocks in the stripe except the particular user data block and the parity data.

[0094] A series of tests may be performed to determine the error block. One test is performed on each of blocks A, B, C, and D. The series of tests is performed on Figure 2 The right side of the diagram shows tests 1 to 4.

[0095] In Test 1, the data of block A is replaced with the sum of A, and the other three blocks B, C, and D have the initially acquired data. A is replaced with the sum of all three remaining data, plus the parity data. The CRC is recalculated after the replacement. In this scenario, the CRC still detects an error in the data stripe because the error is in block C in the initially acquired stripe.

[0096] In Test 2, the data of block B is replaced, and blocks A, C, and D have data from the originally obtained stripe. For the same reasons as described in Test 1, the CRC again indicates an error.

[0097] In Test 3, block C is replaced with other blocks A, B, and D with initially obtained data. In this case, the CRC indicates no error because block C, which had an error in the initially obtained data, has been replaced. This identifies block C as an error block in the initially obtained data stripe.

[0098] In Test 4, block D is replaced, and the other blocks A, B, and C have the originally obtained data. In this scenario as well, the recalculated CRC indicates an error, because the error is in the originally obtained block C.

[0099] During the execution of the series of tests, it will be understood that when an erroneous block is detected, the process can be stopped. In the illustrated series of tests (Test 1, Test 2, Test 3, and Test 4), the series can terminate after Test 3, and block C can be determined to be an erroneous block.

[0100] The testing process described above for detecting specific user data / media components having errors identified by a CRC and recovering from the errors if possible may be referred to below as "multiple testing".

[0101] It should be understood that although multiple tests are described above for an LRAID4+1 configuration, the process can be used to detect and optionally correct data errors in LRAID configurations of various sizes. Many of the examples described in this application use LRAID8+1 and LRAID16+1 configurations for illustration, and multiple tests can be used for each of these configurations to detect and recover data errors.

[0102] Figure 3 An example of a first memory access to data using a first access granularity and a second memory access using a second access granularity for error recovery according to an embodiment of the present disclosure is graphically illustrated.

[0103] The depiction illustrates conventional use of LRAID8+1 302. In conventional use 302, access to a particular set of 16 media components may be achieved through two separate LRAID8+1 requests, each request producing one of two stripes, where each stripe has user data and corresponding parity for the 16 media components.

[0104] Depiction 304 illustrates access to a particular set of 16 memory elements according to an example embodiment. In an example embodiment, the elements of the particular set may be accessed by an LRAID8+1 request to obtain one of the LRAID8+1 stripes (data blocks) 306 or 308 as shown, or by an LRAID16+1 request to obtain stripe (data block) 312 and parity 310. Note that LRAID8+1 may not require parity to be returned, while parity is returned for LRAID16+1 requests.

[0105] An embodiment combines LRAID8+1 and LRAID16+1 capabilities into a two-step data access and error recovery that provides initial data access in LRAID8+1 mode at an access granularity of 256 bytes (e.g., a first access granularity) to obtain one of two 256-byte data blocks 306 or 308 (not necessarily including parity 310). The two data blocks 306 and 308 have 64-byte blocks from each media component (e.g., storage media device 118) that is part of a 16-media device RAID stripe. Parity 310 is stored on the 17th media device in the same stripe. That is, the LRAID16+1 stripe is arranged on 17 media devices, 16 of which are used to store user data, and one media device is used to store parity data for user data striped on the other 16 media devices in the stripe.

[0106] The data request that triggers LRAID8+1 access to the media may be a read request received at the target CXL controller 122 from the host 102. The target CXL controller 122 and / or the central controller portion 114 (e.g., the MML layer 124 in the central controller portion 114) may generate a corresponding data access request to access the storage media device 118. The RAID circuitry 132 controls access to the LRAID to obtain the requested data from the media.

[0107] The data block received from the media in response to the initial read request may be written to a cache line (eg, a 256-byte cache line) in cache 126. If parity information is available, it may also be stored in cache 126.

[0108] For example, a CRC or other check on the retrieved data may be performed by ECC circuitry 130. If an error is detected in regular use 302 obtained in LRAID8+1 mode, a larger granularity LRAID16+1 (eg, a second access granularity) may be used to attempt to recover from the error.

[0109] Therefore, one or more of the component ECC circuitry 130, RAS circuitry 128, and RAID circuitry 132 operate to generate and transmit an LRAID16+1 request to a media device in the RAID array. In response, the LRAID16+1 access obtains a complete stripe of LRAID16+1 array data blocks 312, which includes a 64-byte block from each of the 16 media devices in the RAID array and a parity block from the 17th media device.

[0110] Although the two-step process of the example embodiment is described using the examples of LRAID8+1 and LRAID16+1, the embodiment is not limited thereto.

[0111] Figure 4 A scenario when a first memory access for reading data has no data error is graphically illustrated according to some embodiments of the present disclosure.

[0112] Data 402 illustrates a data stripe including 16 data blocks and parity data stored in corresponding media components. Figure 4 A 512-byte stripe is illustrated to include two 256-byte data sub-stripes, with 64-byte data blocks stored on each of the 16 media components that are part of the RAID stripe. The parity data is the parity of the entire data stripe. That is, the parity is calculated by considering each of the 16 data blocks (e.g., a bitwise XOR of the 16 blocks).

[0113] When a read request is received, in some example embodiments, the cache may identify the media component and / or RAID array where the requested user data is located. The cache may then obtain a cache line (e.g., 256 bytes) of data from the RAID array. According to an embodiment, LRAID8+1 mode access is used to access only the sub-stripe containing the requested user data. Another sub-stripe of the stripe and parity data that is a component of the same stripe are not accessed in the initial read request.

[0114] The requested user data is present in the newly acquired cache line of data 404. The requested user data is indicated in the figure by the two solid filled blocks shown in data on the medium 402 and data in cache 404. The unfilled blocks represent user data other than the requested user data, but in the same stripe or sub-stripe as the requested user data.

[0115] A CRC check is performed on the newly acquired data cache line.

[0116] exist Figure 4 In the illustrated scenario, the CRC test passes and no errors are present in the obtained data 404. Therefore, the obtained data 404 or a portion thereof (eg, user data) may be returned to the requestor (eg, host 102).

[0117] Figure 5 A scenario when a first memory access for reading data has a data error according to some embodiments of the present disclosure is graphically illustrated.

[0118] Data 502 and 504 show an initial read access to a stripe of data 502 in the media, and an identified cache line of data 504 obtained from the media in response to a read request and then stored in the cache. Figure 4 Same explanation.

[0119] exist Figure 5 In the scenario, assume that the CRC check fails, indicating an error in the obtained data cache line. For greater clarity, the "X" indicates an erroneous block and needs to be subsequently identified by the correct operating system. When it is determined that there is an error in the 256 bytes of data obtained from the initial read request in LRAID8+1 mode, the system initiates error recovery. Error recovery includes issuing a read retry for the entire 512 bytes of the stripe in LRAID16+1 mode.

[0120] As a result of the read retry, data 506 is obtained from the media and written to the cache. In an embodiment, only sub-strips that were not previously retrieved are written to the cache because another sub-strip (where an error was detected) is already in the cache. As can be seen, data 506 includes 64 bytes of data for the stripe and 16 blocks of parity data. An "X" indicates a cache line in which there is erroneous data. The parity data in 506 is a parity calculated based on all 16 blocks in the stripe.

[0121] As mentioned above about Figure 2 As described, given a plurality of blocks and associated parity data for the plurality of data blocks, it is possible to obtain the parity data by, for example Figure 2514. Thus, the system calculates the correct parity for the erroneous cache line 516 (i.e., the parity of the cache line if it were not erroneous). The erroneous cache line 516 and the corresponding calculated parity 514 are in Figure 5 is shown as data 508 .

[0122] The calculated correct parity 514 for the erroneous cache line 516 may be calculated by adding the parity of the non-erroneous cache line 512 to the media-obtained RAID parity 510. Since the parity data P510 obtained from the media is the sum of P1 and P2, which are the parities of the first 256 bytes and the second 256 bytes, respectively, of the 512-byte stripe, the parity of only the first 256-byte cache line (the erroneous cache line in the illustrated scenario) may be determined by simply adding P2 to P.

[0123] Once the correct P1 is determined, the Figure 2 to identify error blocks and, in some embodiments, recover from the errors. Figure 4 The same is true in Figure 5 , the requested user data is shown by a solid filled block.

[0124] Figure 6 A diagram is used to illustrate a scenario when the first memory access for writing data does not have a data error according to some embodiments of the present disclosure. Data 602 shows the RAID stripe where the data block to be updated is located. Figure 4 and 5 Likewise, the data blocks to be updated are shown in solid fill. As shown, the data to be updated is located in the first 256 bytes of the stripe (ie, the first sub-strip).

[0125] To update data, the first 256 bytes are read from the media device and written to a cache (e.g., cache 126). In addition, since updates to the data require correction of the associated parity data, the RAID parity data is also obtained from the media and stored in the cache. Figure 6 In the illustrated scenario, the CRC test passes and therefore no errors are detected.Data 604 shows the first sub-stripe of the stripe and the stripe parity stored in the cache.

[0126] The updating of the parity data may be performed by first calculating the difference between the blocks due to the update. The calculated difference between the before and after blocks of user data (i.e., the difference between the identified blocks stored in the media and the identified blocks provided in the user data with the write request) may be used to calculate a parity value, and the parity value may be added to the RAID parity already obtained and stored in the cache.

[0127] The updated parity is the updated parity of the RAID stripe.Data 604 including the user data and the updated RAID parity may then be written from the cache to the media.

[0128] Figure 7 A scenario when a first memory access for writing data has a data error according to some embodiments of the present disclosure is graphically illustrated.

[0129] Data 702 shows the associated RAID stripes in the media, and data 704 shows the cache lines containing the data blocks to be updated with user data in a write operation. A write access to the media obtains the first 256-byte sub-stripe and the RAID parity of the stripe from the stripe. It should be noted that while it is sufficient to obtain only the first sub-stripe with the identified data blocks from the stripe in response to a read request, the RAID parity 710 of the sub-stripe and the stripe is obtained in response to a write request.

[0130] Perform a CRC check and Figure 7 In the illustrated scenario, it is assumed that the CRC check fails. This indicates an error in the first sub-strip obtained now in cache line 716. An "X" indicates an error. Figure 5 The errors are detected and corrected in the same manner as described for the error scenario of the read request shown in . The second sub-stripe 712 and the RAID parity 710 are shown as 706. The error cache line 716 and the correct parity 714 of the error cache line 716 are shown as 708.

[0131] After the error is corrected, the delta of the updated data block can be calculated, and the updated RAID parity can be calculated, such as Figure 6 This is described by the error-free scenario of the write request shown in . Figure 6 The updated cache line and parity are written back to the media in the same manner as described.

[0132] Figure 8A memory device in a locked RAID (LRAID) arrangement for LRAID16+1 with 2 LRAID8+1 sub-stripes is illustrated in accordance with some embodiments of the present disclosure. In the illustrated module 800, it has a left side and a right side, wherein each of the left and right sides includes 4 rows of 17 die (16 data die and 1 parity die) stripes. With 1 parity die used for every 16 data die, the parity overhead is 6.25%.

[0133] The left side of module 800 includes an LRAID16+1 array 808 comprised of two LRAID8+1 arrays 802 and 804 and parity 806. The parity 806 in each row represents the parity of that row across all the dies included in LRAID8+1 802 and 804. The right side of module 800 includes an LRAID16+1 array 818 comprised of two LRAID8+1 arrays 812 and 814 and parity 816. The parity 816 in each row represents the parity of that row across all the dies included in LRAID8+1 812 and 814.

[0134] Each of the LRAID8+1 802, 804, 812, and 814, each of the parity 806 and 816, and each of the LRAID16+1 808 and 818 are individually accessible from a memory device, such as, for example, memory device 104. In an embodiment, each LRAID8+1 array 802, 804, 812, and 814 maps to a respective 64-bit channel on the memory device, and each parity die array maps to a respective 8-bit channel on the memory device.

[0135] Fig. 9 Memory devices in an LRAID arrangement for LRAID32+1 with 4 LRAID8+1 sub-stripes according to some embodiments of the present disclosure are described.

[0136] The illustrated module includes 4 ranks of 33 dies (32 data dies and 1 parity die). As with module 800, in module 900, each LRAID4+1 array, parity die array, and LRAID16+1 array may be individually accessed from a memory device, such as memory device 104, for example. Each of LRAID8+1 arrays 902, 904, 906, and 908 may be mapped to a respective 64-bit channel, and the parity die array may be mapped to an 8-bit channel.

[0137] The parity overhead in module 900 is 1 parity die for 33 data dies to produce a result of 3.25%, which is half the overhead cost of module 800. Thus, example embodiments may utilize different configurations of user data blocks and parity data blocks in an LRAID stripe that produce different overhead cost levels.

[0138] Fig.10 Flowchart illustrating a process performed at a target CXL controller for a read request according to some embodiments of the present disclosure. Process 1000 begins when a memory controller receives a read request from a requester (eg, host 102) at operation 1002. At operation 1002, target CXL controller 122 receives the read request.

[0139] At operation 1004, the target CXL controller 122 and / or the central controller portion 114 identifies the RAID array containing the requested user data (e.g., identifies a substripe / stripe in the RAID array) and generates a memory read command. The RAID array may be identified based on the address of the requested user data or the requested block of user data.

[0140] At operation 1006, the RAID circuitry 132 accesses the identified RAID array in LRAID8+1 mode (i.e., 256-byte access granularity) to obtain a sub-stripe (e.g., a 256-byte sub-stripe) of user data containing the requested user data from the LRAID16+1 stripe. The LRAID8+1 sub-stripe contains data stored on 8 media components. Each of the 8 media components contained in the LRAID8+1 sub-stripe and storing user data provides 32 bytes of the 256-byte user data obtained by the central controller portion 114.

[0141] At operation 1008, the CRC is checked (i.e., a CRC test is performed) against the obtained user data to determine if there are errors in the obtained user data. Note that in some embodiments, the CRC information is stored as metadata in the media component and / or cache 126. If the CRC check passes (i.e., there are no errors), then at operation 1010, the data is sent to the host (or other source from which the read request was received).

[0142] If the CRC check fails (i.e., there is an error), then error recovery is initiated and a read retry is generated and sent for the desired user data using LRAID16+1 (e.g., 512-byte access granularity) at operation 1012. Assume that the cache line storing the accessed LRAID8+1 sub-stripe user data (i.e., the cache line where the detected error exists) is CLi.

[0143] As described above, although the CRC check of the obtained LRAID8+1 user data detects the presence of errors in the obtained user data, it does not identify which of the several memory components in the stripe provided the erroneous user data. Therefore, the process 1000 performs an error recovery process to identify and then correct the errors in the user data if possible.

[0144] At operation 1014, the user data obtained in response to the read retry is uploaded to a cache (e.g., cache 126). Thus, the entire LRAID16+1 stripe (including parity) is written to the cache. Thus, 32 bytes of user data and an additional 32 bytes of parity data from each of the 16 memory components storing user data may be written to cache 126. The LRAID16+1 parity may be referred to as P.

[0145] Note that P=Pi+Pj, where P, Pi, and Pj are the parity of the LRAID16+1 parity, the LRAID8+1 parity of the cache line CLi, and the LRAID8+1 parity of the cache line CLj. At operation 1016, the parity Pi used in the multiple test of CLi is calculated by adding the entire (correct) CLj to the LRAID16+1 parity P, i.e.: Pi=P+CLj. Figure 2 An example of multiple testing according to example embodiments is described.

[0146] At operation 1018, the error location is identified and CLi is corrected. Figure 2 Example operations for identifying and correcting errors in a user data block (i.e., user data obtained from a media component of an LRAID array) are described for an LRAID4+1 configuration. According to an embodiment, operation 1018 performs a similar series of tests for each user data block in CLi. Note that CLi is user data from an LRAID8+1 stripe. As described with respect to Figure 2 As described for the scenario of LRAID4+1, in this example, in the worst case, 8 tests (i.e., tests for each of the 8 user data blocks) are performed serially or in parallel or in combination thereof, where each test first replaces the corresponding user data block with the XOR sum of all other user data blocks and the parity Pi. Figure 2 As described, the multiple test process also recovers from errors in identified user data blocks.

[0147] At operation 1020, the requested user data is provided to the host (or other source from which the read request was received). Note that at operation 1014, before using the obtained recovery data, process 1000 may check the CRC of CLj to improve reliability. If CLj passes the CRC check, then error recovery proceeds as described above. If CLj fails the CRC check, then it is a scenario where two errors were detected in a single LRAID16+1 stripe, and the errors may be unrecoverable. In some embodiments, when the CXL controller or central controller portion 114 identifies an unrecoverable error, it notifies the host 102 by sending a CXL RAS message indicating "poisoning" so that the host can take action as needed.

[0148] Note also that we can use only LRAID16+1 accesses to perform read accesses and read error recovery. However, in the event of an error in an LRAID16+1 stripe, multiple tests for identifying erroneous user data blocks / memory components will have a worst case of 16 (rather than the worst case of 8 for LRAID8+1). Therefore, the combination of LRAID8+1 and LRAID16+1 as in the example embodiment provides improved performance in terms of error recovery compared to using only LRAID16+1 accesses to perform read accesses and read error recovery.

[0149] Fig.11 Flowchart illustrating a process performed at a target CXL controller for a write request according to some embodiments of the present disclosure. In some embodiments, process 1100 is performed to write data. Writing user data (which updates user data in a storage media component in response to a write request) requires updating the CRC of the affected (i.e., updated) LRAID8+1 sub-stripe and updating the parity of the corresponding entire LRAID16+1 stripe according to the update.

[0150] At operation 1102, a write request is received at the target CXL controller 122. At operation 1104, a corresponding read command is generated by the target CXL controller 122 and / or the central controller portion 114 to the media component in LRAID8+1 mode to obtain the LRAID8+1 stripe containing the user data to be updated according to the received write request. The generation of the read command may require the target CXL controller 122 and / or the central controller portion 114 to determine the LRAID8+1 sub-stripe in which the user data is to be updated. Fig.10 In contrast to the read operation described in , in this process 1100 , LRAID16+1 stripe parity is also obtained.

[0151] At operation 1106, the obtained user data of the LRAID8+1 sub-stripe is stored in the cache 126. In some embodiments, the obtained user data of the LRAID8+1 sub-stripe is stored in one cache line in the cache 126. Fig.10 In contrast to the read operation described in , in this process 1100, the LRAID16+1 stripe parity is also obtained and stored in the cache 126. Note that since the write operation requires the LRAID16+1 parity to be updated as well, when the user data of the LRAID8+1 sub-stripe is uploaded to the cache 126, the parity data from the LRAID16+1 stripe is also uploaded to the cache (e.g., cache 126). Note that in some embodiments, considering that the two access granularities are LRAID8+1 and LRAID16+1, the media component only stores the parity P for the LRAID16+1 stripe, and does not store the separate parities (Pi and Pj) for the two LRAID8+1 sub-stripes in the LRAID16+1.

[0152] At operation 1108, a CRC check is performed on the obtained sub-stripe of user data. In an example embodiment, the CRC is calculated for a cache line, which in some embodiments is a typical granularity of access to the media (e.g., 256 bytes of user data corresponding to the LRAID8+1 configuration described in this application). As described above, in some embodiments, the CRC of the LRAID8+1 sub-stripe may be stored / maintained as metadata in association with the corresponding user data in a reserved area in the media and / or in the cache line. This operation is similar to the CRC check performed in the read process 1000.

[0153] At operation 1110, if the CRC check fails, identification and correction of erroneous user data blocks is performed in the same manner as described with respect to operations 1012 through 1018 of process 1000 by issuing a read retry for the LRAID16+1 stripe to obtain the stripe and stripe parity P (or by obtaining only the second sub-stripe since the first sub-stripe and stripe parity P are already in cache). In some embodiments, the RAID parity is uploaded to the cache during the read retry of this step only if the parity is not already in cache. This avoids stressing the media components that store the parity data instead of the data die. One skilled in the art will appreciate that excessive access to memory components can lead to errors.

[0154] At operation 1112, when the CRC is determined to be correct (either based on the check at operation 1108 or after error detection and correction at operation 1110), the user data is updated in cache 126. Since the CRC is in the cache line, the CRC for the updated cache line / sub-stripe is updated. The CRC may simply be updated to account for the user data updated by the received write request.

[0155] At operation 1114, the LRAID16+1 stripe parity is updated. Note that when a LRAID8+1 sub-stripe is uploaded to cache for a write operation, the corresponding LRAID16+1 stripe parity is also uploaded. The LRAID16+1 parity is updated by adding the delta data to the existing parity.

[0156] At operation 1116, the updated sub-stripe and updated parity are written back from cache 126 to the corresponding media component, and optionally, an acknowledgement is transmitted to the host or other entity that sent the write command. This completes the write operation.

[0157] Some embodiments may include ECC encoding the data in response to the memory controller receiving a request from the host to write the data to the DRAM component. The data may be ECC encoded so that the data can be recovered even if there are errors in the data. In the same embodiment, the data may be ECC encoded before writing the data to the DRAM memory device. Some embodiments include encrypting the data. The encryption may be AES encryption. In some embodiments, the data may be ECC encoded before the data is encrypted. The data may be encrypted by a security component on the memory controller.

[0158] The above-mentioned embodiments of the present disclosure improve certain aspects of conventional memory access technologies (for example, such as LRAID8+1 and LRAID16+1). The method adopted in the embodiments utilizes the performance advantage of LRAID8+1 and the error protection advantage of LRAID16+1.

[0159] As described above, in response to a typical read request or write request sent from a host to a memory device in a CXL environment for a 64-byte block of user data, the LRAID system accesses the entire stripe in the back-end media component. Although the bandwidth observable on the host-memory device interface corresponds to the 64 bytes of user data involved in the read or write, the back-end bandwidth consumption in the LRAID system is significantly greater, i.e., 64 bytes vs. 256 bytes in LRAID8+1 and 64 bytes vs. 512 bytes in LRAID16+1. Therefore, in both access granularities, LRAID8+1 provides better performance (e.g., 256 bytes retrieved from the media compared to 512 bytes retrieved in LRAID16+1), and LRAID16+1 provides better bandwidth cost in the back end (i.e., lower cost; 32-byte parity for 512 bytes of user data (cost 6.5%) compared to 32-byte parity for 256 bytes of user data (cost 12.25%)).

[0160] Described in another way, certain embodiments provide the performance of an LRAID8+1 solution while reducing its bandwidth consumption cost at the back-end-media interface compared to conventional systems. Certain embodiments use LRAID8+1 granularity to access media components without retrieving RAID parity data, and use LRAID16+1 granularity to recover errors detected in LRAID8+1 user data. In order to improve the efficiency of the technology, an example embodiment configures the RAID array so that starting from two LRAID8+1 stripes, the parity of the corresponding LRAID16+1 stripe is the sum of the parities of the two LRAID8+1 stripes. In addition, in order to further improve the efficiency of the technology, since the most frequent access in the usage scenario is at the LRAID8+1 access granularity, each of the two constituent LRAID8+1 stripes of the larger LRAID16+1 stripe has its own CRC.

[0161] Since the occurrence of errors is rare compared to data access, certain embodiments combine the advantages of both access granularities by using LRAID8+1 access granularity (i.e., 256 bytes) to access data without retrieving RAID parity, and then, only when an error is detected, using a larger access granularity (i.e., 512 bytes) together with RAID parity to manage the error starting with retrying the access.

[0162] Although all accesses to the media are locked in the example embodiment, the lock domain is changed according to the purpose of the access. For example, in the example embodiment of accessing in LRAID8+1 mode or LRAID16+1 mode with a burst length of 8 bytes and an access block size of 1 byte, the lock domain may have different byte values ​​for different types of access: 8 bytes for read access, which is equivalent to locking one byte from each of the 8 media components in the LRAID8+1 stripe; 9 bytes for write access, which is equivalent to locking each of the 8 media components storing user data and the corresponding parity media components in the LRAID8+1 stripe; and 17 bytes for error recovery access in the LRAID16+1 stripe.

[0163] The example embodiments are described above primarily with respect to the CXL 2.0 and / or 3.0 specifications. Furthermore, some of the described embodiments describe a CXL type 3 memory device communicating with a host. However, it should be understood that the embodiments are not limited to the CXL or CXL versions specifically mentioned herein. Furthermore, the example embodiments may be applicable to devices other than memory devices.

[0164] While the invention has been described in connection with what are presently considered to be the most practical and preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but on the contrary is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0165] It should be noted that the methods described above describe possible implementations, and the operations and steps may be rearranged or otherwise modified, and other implementations are possible. In addition, parts from two or more of the methods may be combined.

[0166] For example, the various illustrative blocks and components described herein in connection with the present disclosure may be implemented or performed using circuit systems designed to perform the functions described herein, such as a general purpose processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any processor, controller, microcontroller, or state machine. A processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0167] Computer-readable media include both non-transitory computer storage media and communication media, including any media that facilitates the transfer of computer programs from one place to another. Non-transitory storage media can be any available media that can be accessed by a general or special purpose computer. As an example and not limitation, non-transitory computer-readable media may include RAM, ROM, electrically erasable programmable read-only memory (EEPROM), compact disk (CD) ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other non-transitory media that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a general or special purpose computer or a general or special purpose processor. Moreover, any connection is properly referred to as a computer-readable medium.

[0168] The description herein is provided to enable one skilled in the art to make or use the present disclosure. Various modifications to the present disclosure will be apparent to one skilled in the art, and the general principles defined herein may be applied to other variations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but should be in the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A memory access method, comprising: In response to a storage access request, obtaining a sub-stripe from a data stripe stored on a plurality of storage media components arranged in a redundant array of independent data RAID using a first access granularity; detecting errors in the sub-strip of the data; and In response to the detecting the error, the stripe of the data is obtained from the plurality of storage media components using a second access granularity, wherein the second access granularity is greater than the first access granularity.

2. The memory access method according to claim 1, further comprising: The location of the error is identified in the sub-stripe using the stripe obtained using the second access granularity and the parity information of the stripe.

3. The memory access method according to claim 1, further comprising: Send via Compute Express Link TM CXL TM A message is used to notify the sender of the memory access request about the detected error. 4 . The memory access method of claim 1 , wherein the CXL message complies with reliability, availability, and serviceability (RAS) capabilities of the CXL protocol. 5 . The memory access method of claim 1 , wherein the method further comprises storing the sub-stripe in a cache, and wherein the first access granularity or the second access granularity corresponds to a size of a cache line of the cache.

6. The memory access method according to claim 1, wherein the memory access request is a read request or a write request, and wherein when the memory access request is a write request, the obtaining of the sub-stripe using the first access granularity includes obtaining the user data blocks of the sub-stripe and the parity information of the stripe, and when the memory access request is a read request, the obtaining of the sub-stripe using the first access granularity includes obtaining the user data blocks of the sub-stripe without obtaining the parity information of the stripe.

7. The memory access method of claim 1, wherein the number of memory media components of the RAID accessed using the first access granularity is less than the number of memory media components of the RAID accessed using the second access granularity.

8. The memory access method according to claim 7, wherein the number of memory media components of the RAID accessed using the second access granularity is n times the number of memory media components of the RAID accessed using the first access granularity, wherein, n is a multiple of 2.

9. The memory access method of claim 1, wherein the detecting the error comprises detecting the error based on error correction information associated with the sub-stripe.

10. An apparatus comprising: a host interface comprising a plurality of input / output (I / O) pathways configured to couple to a host device; a physical PHY layer interface configured to couple to a plurality of storage media components; and A control circuit system configured to perform operations including: In response to a memory access request received through the host interface, obtaining a sub-stripe from a data stripe stored on the plurality of memory media components arranged in a redundant array of independent data (RAID) using a first access granularity and the PHY layer interface; detecting errors in the sub-strip of the data; and In response to the detecting the error, the stripe of the data is obtained from the plurality of storage media components using a second access granularity and the PHY layer interface, wherein the second access granularity is greater than the first access granularity.

Images