Fault tolerance in distributed memory systems

Abstract

Methods, systems, devices, and architectures are disclosed in some examples that provide techniques for memory devices and memory fabric redundancy within a distributed memory system. In some examples, two memory devices are paired and each store the same data set such that writes to the memory devices are replicated and reads from either device may be satisfied. In some examples, a memory processing unit (MPU) may be incorporated into a memory architecture to support these paired memory devices. The MPU may be placed between the host and a multi-plane memory fabric connected to a multi-port CXL memory device. In some examples, the MPU may also enable use of an alternative fabric link. That is, if a memory fabric link between the MPU and a memory device is unavailable, a connection to the memory device may be resumed with an alternate link.

Description

[0001] Government Rights

[0002] This invention was made with U.S. Government support under Contract No. DE-AC05-76RL01830 to Pacific Northwest National Laboratory. The U.S. Government has certain rights in this invention. Technical Field

[0003] Embodiments relate to fault tolerance in a distributed memory architecture. Some embodiments relate to using redundant memory devices in a distributed memory architecture. Some embodiments relate to using redundant fabrics in a distributed memory architecture. Background Art

[0004] Memory devices used in computers or other electronic devices can be classified as volatile and non-volatile memory. Volatile memory requires power to maintain its data and includes random access memory (RAM), dynamic random access memory (DRAM), or synchronous dynamic random access memory (SDRAM), etc. Non-volatile memory can retain stored data when power is not supplied, and includes flash memory, read-only memory (ROM), electrically erasable programmable ROM (EEPROM), static RAM (SRAM), erasable programmable ROM (EPROM), resistance variable memory, phase change memory, storage class memory, resistive random access memory (RRAM), and magnetoresistive random access memory (MRAM), etc. Persistent memory is an architectural property of a system, where data stored in the media is available after a system reset or power cycle. In some examples, non-volatile memory media can be used to build a system with a persistent memory model.

[0005] A memory device may be coupled to a host (e.g., a host computing device) to store data, commands, and / or instructions for use by the host when the computer or electronic system is operating. For example, during operation of a computing or other electronic system, data, commands, and / or instructions may be transferred between the host and the memory device(s).

[0006] Various protocols or standards may be applied to facilitate communication between a host and one or more other devices (such as a memory buffer, accelerator, or other input / output device). In an example, a protocol such as Compute Express Link (CXL) may be used to provide a high bandwidth and low latency connection. Summary of the invention BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In the drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar components in different views. The same reference numerals with different letter suffixes may represent different instances of similar components. The drawings generally illustrate the various embodiments discussed in this document by way of example and not by way of limitation.

[0008] Figure 1 Schematic diagram showing system topology with availability improvement hardware according to some examples of the present disclosure.

[0009] Figure 2 A message passing diagram showing a read request following a normal path and serviced by a master CXL device according to some examples of the present disclosure.

[0010] Figure 3 A message passing diagram of a write request according to some examples of the present disclosure is shown.

[0011] Figure 4 A message passing diagram illustrating a read request serviced by a secondary CXL device according to some examples of the present disclosure.

[0012] Figure 5 A flow chart illustrating device replacement according to some examples of the present disclosure.

[0013] Figure 6 A sequence diagram illustrating fabric manager involvement in device replacement according to some examples of the present disclosure is shown.

[0014] Figure 7 A diagram showing an MPU positioned within a multi-level memory fabric switch.

[0015] Figure 8 A diagram showing a mirroring and active fabric fault tolerance engine 814 according to some examples of the present disclosure.

[0016] Fig. 9 A flow chart showing a method for mirroring and active fabric fault tolerance in a distributed memory system according to some examples of the present disclosure.

[0017] Fig.10 Diagram showing roles for removing and replacing non-functional devices according to some examples of the present disclosure.

[0018] Fig.11 is a block diagram illustrating an example of a machine upon which one or more embodiments may be implemented. DETAILED DESCRIPTION

[0019] Compute Express Link (CXL) is an open standard interconnect configured for high-bandwidth, low-latency connections between host devices and other devices such as accelerators, memory devices, and intelligent I / O devices. CXL is designed to facilitate high-performance computing workloads by supporting heterogeneous processing and memory systems. CXL implements coherency and memory semantics on top of PCI Express (PCIe)-based I / O semantics to optimize performance.

[0020] In some instances, CXL is used in applications such as artificial intelligence, machine learning, analytics, cloud infrastructure, edge computing devices, communication systems, and elsewhere. Data processing in such applications can use a variety of scalar, vector, matrix, and spatial architectures that can be deployed in CPUs, GPUs, FPGAs, smart NICs, and other accelerators that can be coupled using CXL links.

[0021] CXL supports dynamic multiplexing using a set of protocols that include input / output (CXL.io, based on PCIe), cache (CXL.cache), and memory (CXL.memory) semantics. In an example, CXL can be used to maintain a unified, consistent memory space between a CPU (e.g., a host device or host processor) and any memory on an attached CXL device. This configuration allows the CPU and other devices to share resources and operate on the same memory region for higher performance, reduced data movement, and reduced software stack complexity. In an example, the CPU is primarily responsible for maintaining or managing consistency in a CXL environment. Therefore, CXL can be utilized to help reduce device cost and complexity, as well as the overhead traditionally associated with consistency across I / O links.

[0022] CXL runs on top of the PCIe PHY and provides full interoperability with PCIe. In an example, if the link partner of a CXL device is capable of supporting CXL, the CXL device begins link training at the PCIe Gen 1 data rate and negotiates CXL as its operating protocol (e.g., using the alternative protocol negotiation mechanism defined in the PCIe 5.0 specification). As a result, devices and platforms can more easily adopt CXL by leveraging the PCIe infrastructure without having to design and validate PHYs, channels, channel extension devices, or other upper layers of PCIe.

[0023] In an example, CXL supports single-stage switching to achieve fan-out to multiple devices. This enables multiple devices in a platform to be migrated to CXL while maintaining CXL's backward compatibility and low latency characteristics.

[0024] In an example, CXL can provide a standardized computing fabric that supports multiple logical devices (MLDs) and pooling of a single logical device, such as using a CXL switch connected to several host devices or nodes (e.g., root ports). This feature enables servers to pool resources, such as accelerators and / or memory, that can be assigned based on workload. For example, CXL can help facilitate resource allocation or dedication and release. In an example, CXL can help allocate and deallocate memory to various host devices as needed. This flexibility helps designers avoid over-provisioning while ensuring optimal performance. The CXL protocol enables the construction of large, multi-host, fabric-attached memory systems. In addition, CXL memory systems can be built from multi-port hot-swappable devices and connected to hot-swappable memory switches.

[0025] CXL 3.0 may give fabric-attached memory devices the ability to provide shared memory that can be read and written by multiple hosts.

[0026] Some of the compute-intensive applications and operations mentioned herein may require or use large data sets. Memory devices storing such data sets may be configured for low latency and high bandwidth as well as persistence. One problem with load-store interconnect architectures involves ensuring persistence. CXL can help solve this problem using architectural flows and standard memory management interfaces for software, for example enabling the movement of persistent memory from a controller-based approach to direct memory management.

[0027] One problem with large-scale distributed memory systems is reliability. As the number of components in these systems increases, the number of failure points also increases. For large memory systems, the chance of media failure on a particular memory device increases. In addition to media failures, many errors may not be directly associated with media failures. For example, many failures may be associated with power delivery systems, memory fabrics, cables, and other miscellaneous problems. Modern microprocessors cannot properly handle memory read request failures. In the best case, applications whose load instructions are not satisfied within a short time frame are terminated. In many cases, all processes and OS running on the server are terminated, causing the server to restart. This can compromise the availability of servers using memory modules and memory fabrics.

[0028] In some examples, methods, systems, devices, and architectures are disclosed that provide techniques for memory device and memory fabric redundancy within a distributed memory system. In some examples, two memory devices are paired and each stores the same data set so that writes to the memory devices are replicated and can satisfy reads from either device. In some examples, a memory processing unit (MPU) can be incorporated into the memory architecture to support these paired memory devices. The MPU can be placed between a host and a multi-plane memory fabric connected to a multi-port CXL memory device. In some examples, the MPU can also enable the use of an alternate fabric link. That is, if the memory fabric link between the MPU and the memory device is unavailable, an alternate link can be utilized to restore the connection to the memory device.

[0029] Figure 1 A schematic diagram of a system topology 100 with availability improved hardware according to some examples of the present disclosure is shown. A host system 105, which may include one or more processors, may be connected to a memory processing unit (MPU) 110. The MPU 110 may be part of the host system 105, or may be connected to the host system 105 through a communication interface. The MPU 110 may include a host interface 112 for communicating with the host system 105, a memory mirroring and active fabric fault tolerance engine 114 for implementing fault tolerance techniques such as those disclosed herein, and a fabric interface 116 for interfacing with a memory fabric. Fabric plane A 120 and fabric plane B 118 form redundant fabric links to multiple multi-port CXL devices 122, 124, and 126. A fabric manager 128 manages the memory fabric. In Figure 1 In FIG. 1 , dashed connections represent CXL links over the CXL.mem protocol. In some examples, MPU 110 may be or include one or more hardware devices, such as a processor, memory, application specific integrated circuit (ASIC), custom silicon, or the like.

[0030] In some examples, in order to allow the host processor to continue to operate regardless of memory module problems, the memory mirroring and active fabric fault tolerance engine 114 mirrors all write requests to two different memory devices. These devices are referred to herein as a memory device pair consisting of a memory device A and a memory device B. In contrast to write requests sent to the two devices in the pair, read requests are first served by the master device, and the auxiliary device is not utilized unless the master device does not respond or is unavailable. By dividing the memory devices into master devices and auxiliary devices, this increases the memory bandwidth used for servicing read requests during normal operation. That is, two memory reads targeting different physical devices of the memory device pair can be served simultaneously or nearly simultaneously. Various methods can be used to select which device in the pair is the master device and which is the auxiliary device. In some examples, one or more address bits can be used to determine the master address. For these examples, some addresses select device A as the master device, while other addresses select device B as the master device. In some examples, the bit on the most significant bit of the host physical address is used to determine whether the master device (i.e., the device that first services the read request) is device A or device B in the memory device pair.

[0031] As described above, a read request may be first sent to the primary device, and if the response fails, the read request may be forwarded to the secondary device. Figure 2 Message passing diagram 200 shows a read request that follows a normal path and is serviced by primary CXL device 222, according to some examples of the present disclosure. Host 205 sends a MEMRD (memory read) request to MPU 210. MPU 210 determines which CXL device is the master device corresponding to the request. For example, using the address of the request. MPU 210 then forwards the request to primary CXL device 222. In some examples, the MPU sets a timer that is less than the timeout value of host 205. If the timer expires, MPU 210 may retry the request (e.g., over an alternate fabric link) or send the request to secondary CXL device 224. Figure 2 In the example of FIG. 2 , master CXL device 222 responds with data, which is then forwarded back to host 205 by MPU 210 .

[0032] Figure 3 A message passing diagram 300 of a write request according to some examples of the present disclosure is shown. The host 305 sends a memory write request MEMWR to the MPU 310, which forwards the write request to both the primary CXL device 322 and the secondary CXL device 324. Each device provides a completion message CMP back to the MPU. Upon receiving both CMP messages, the MPU 310 may forward the CMP back to the host 305.

[0033] In some examples, as previously described, the system may employ alternate paths and / or redundant fabrics to avoid failed fabric components and links. Thus, in some examples, a read or write command that does not complete successfully may time out, and in response, the MPU may try an alternate path or redundant fabric. In some examples, for a read request, the MPU may try an alternate path and / or redundant fabric before trying a secondary device.

[0034] Figure 4 Message passing diagram 400 of a read request serviced by secondary CXL device 424 is shown according to some examples of the present disclosure. Host 405 sends a MEMRD (memory read) request to MPU 410. MPU 410 determines which CXL device is the master device corresponding to the request. For example, using the address of the request. MPU 410 then forwards the request to primary CXL device 422. In some examples, the MPU sets a timer that is less than the timeout value of host 405. If the timer expires, MPU 410 may retry, for example, by utilizing an alternate path or redundant fabric. In these examples, MPU 410 resets the timer. Upon a timeout (not shown) or disconnect notification (as shown), MPU 410 may send a MEMRD to secondary CXL device 424. MPU 410 then forwards a response (MEMDATA) to host 405. If secondary CXL device 424 fails to return data, MPU 410 may send an error, such as a virus, to the host.

[0035] The following is example pseudocode for the read request algorithm performed by the MPU:

[0036]

[0037]

[0038] The pseudo code starts at line 1 after a memory read request is sent to the primary CXL device. At line 1, if the response from the primary CXL device is a "virus" response indicating an error, then the MPU issues a request to the secondary device at line 2. At line 3, if the device returns a virus, then the MPU returns the virus to the host at line 4. Otherwise, at line 6, if the secondary device is disconnected or times out, then the MPU reissues the request to the second device over the alternate fabric link at line 7. If the alternate fabric link is disconnected or times out again, the MPU will return the virus at line 9.

[0039] If the first device does not respond with a virus, but instead returns a disconnect or times out, then the MPU reissues the request to the first device over the alternate fabric link at line 15. If the attempt reports a disconnect or a second timeout at line 16, then the MPU tries the secondary device and issues a failover at line 18. At line 19, if the secondary device is disconnected or times out, then a virus is returned at line 20. In the pseudo code above, if at any point either the primary or secondary device returns good data, then that data is returned to the host.

[0040] The pseudo code for the write command is as follows:

[0041]

[0042] If the write command returns a disconnected state or times out, the MPU attempts an alternate fabric link. If it fails due to a disconnect or a second timeout, the MPU will fail over to a secondary device if the failed device is the primary device.

[0043] The various timeouts may be specifically configured to be larger than the queuing delay of the PCIe link and the link retrain event, but smaller than the PCIe device timeout and the processor last level cache timeout. For example, the queuing delay may be a few microseconds, the link retrain event may be between 5 microseconds and 75 microseconds, the fabric failover timeout may be 10ms, the mirror timeout may be 20ms, and the second mirror timeout may be 30ms. The PCIe device timeout may be 50ms, and the processor last level cache timeout may be 100ms. In some examples, fabric timeouts may be precisely controlled using a per-request timeout count bit and a single global timer count with .1 microsecond resolution. By appropriately sizing the timeouts, the disclosed failure mechanism may operate successfully without triggering a host timeout, but also without being interrupted by link retraining and other tasks.

[0044] Figure 5 Flowchart 500 shows device replacement according to some examples of the present disclosure. In some examples, to enable replacement of CXL devices by relatively slow fabric managers, the MPU may have independent read and write targets to allow online replacement of memory devices without causing software timeouts. This process is described in detail in the following sections. Figure 5. At operation 502, a CXL memory device becomes active and the system begins operation. At operation 504, data mirroring may be activated. Data mirroring may be activated for one or more specific memory addresses determined by an administrator, software application, or the like. At operation 506, the MPU may determine whether any event has indicated a need to replace one or more mirrored CXL memory devices. Examples include module failover indication, virus response, timeout response, operator signaling intent to replace a device (e.g., via a UI), or the like. If there is no replacement event, the system maintains a stable state at operation 504. If there is a replacement event, then at operation 508, the system updates a read target. For example, the MPU updates an address table defining a memory device pair to indicate that one of the memory device pairs to be replaced is unavailable. At operation 510, this process is repeated for a write target. At this point, at operation 512, all read requests and write requests target an operational device in the pair, and replacement of a non-operating device may begin.

[0045] At operation 514, the operator may remove the non-functional device. At operation 516, the operator may install a new device. At operation 518, the new device may be configured. At operation 520, one or more encryption keys of one or more tenants may be unlocked. At operation 522, the new device may be placed in service and the write target may be updated to include the newly placed in service device. At this point, the MPU causes the data from the operating device to be copied to the newly installed device. This may be accomplished at operation 524 by reading all data from the operating device, setting the modification bit for the data, and refreshing all locations. This causes the MPU to write data to both memory devices. At operation 526, once the data is consistent between the two devices, the read target is updated.

[0046] Figure 6 A sequence diagram 600 illustrating fabric manager involvement in device replacement is shown according to some examples of the present disclosure. Figure 6 Corresponds to Figure 5 . The fabric manager notifies the MPU at 610 that a particular CXL device is being shut down. The fabric manager may determine a list of one or more MPUs that service the device being shut down. At 612, the fabric manager changes the read target configuration of the first MPU in the list. At operation 614, the fabric manager changes the read target configuration of the last MPU in the list. At this point, mirrored reads of the out-of-service device are redirected to the other device in the partner pair. At operation 616, the fabric manager changes the write target of the first MPU in the list. At operation 618, the fabric manager changes the target configuration of the last MPU in the list. At this point, the system waits for all old writes and reads to complete, and the out-of-service device can be replaced. After the device is complete, and while Figure 6Not shown in, but in Figure 5 As shown in , the write target of the MPU is updated first, then all data is read, modified, and then refreshed, followed by the read target update.

[0047] The disclosed techniques also operate with multi-level memory fabric switching. Figure 7 Diagram 700 shows an MPU positioned within a multi-level memory fabric switch. Figure 7 As shown, memory fabric switch 718 includes a switch core and one or more upstream switch ports (USPs) and downstream switch ports (DSPs). Hosts (e.g., hosts 702, 704, 706, and 708) connect to MPUs 710, 712, 714, and 716. The MPUs, in turn, connect to USPs in one or more memory fabric leaf switches (e.g., memory fabric leaf switches 720, 721, 722, and 723). Like memory fabric switch 718, the memory fabric leaf switches include a CXL switch core, USPs, and DSPs. Figure 7 As shown, MPUs (e.g., MPUs 710, 712, 714, and 716) are positioned between the first upstream switch port (USP) on the memory fabric and the host. CXL devices 732, 726, and 728 are connected to the DSPs of memory fabric leaf switches 720 and 721, respectively. CXL devices 726 and 728 are in mirrored pair 724. In some examples, write copy units (WDUs) 750, 752 may be installed at the last leaf switch. This may save some fabric bandwidth by performing write copies further down in the fabric. In some examples, this may be achieved by placing the mirrored pairs on the same last-stage leaf switch, such as Figure 7 As displayed.

[0048] Figure 8 A diagram of a mirror and active fabric fault tolerance engine 814 according to some examples of the present disclosure is shown. A CXL host interface 812 sends both a request (REQ) and a request with data (RwD) from the host to the mirror and active fabric fault tolerance engine 814 of the MPU via a receive component (REC) 820. For example, a write request (e.g., with data), a read request, or the like. The mirror and active fabric fault tolerance engine can then process the REQ or RwD appropriately. For example, for a write request, the request can be copied and sent to two CXL device pairs via the CXL fabric interface 816. For a read request, the mirror and active fabric fault tolerance engine determines which device in the device pair will service the read request (e.g., based on the address). If the CXL fabric interface 816 reports that the device is unreachable, an alternative fabric, route, or link can be tried. Responses, data responses (DRS), no data responses (NDR) can be received by the mirror and active fabric fault tolerance engine and processed according to Figures 2 to 6The mirrored and active fabric fault tolerance engine may forward these responses to a response forwarder (RSPFWDER) component 822, which forwards them to the host via the CXL host interface 812. The mirrored and active fabric fault tolerance engine 814 may also communicate with the fabric manager via the fabric management interface 818.

[0049] Mirroring can be expensive to implement. To mitigate this cost, a mechanism can be provided for controlling the mirroring policy and memory fault signaling policy on a 1GB page basis. Large memory systems are often forced to use large pages to achieve good processor performance. In addition, large pages enable the page policy table to reside in on-die memory without compromising the maximum memory size supported by CXL (2 52 The following table compares the various modes that can be enabled for each page in a CXL system:

[0050]

[0051] In some instances, a software recovery strategy can be defined where instead of returning a POISON signal (which normally resets the host) upon failure, the system can return a specified "bad" value or error code. The bad value or error code is then handled using exception handling in the software application. While this has better availability than a standard system because a virus can no longer terminate the host, this solution requires changes to the application software, which the MPU and mirror memory do not.

[0052] Fig. 9A flow chart of a method 900 for mirroring and active fabric fault tolerance in a distributed memory system according to some examples of the present disclosure is shown. In some examples, the method 900 may be performed at an MPU connected to a host across a host interface and connected to a memory device through a memory fabric interface. At operation 910, the MPU may receive a write request to a first memory location. At operation 912, the MPU may identify that the first memory location is a mirrored memory location corresponding to a pair of memory devices including a first memory device and a second memory device. For example, this determination may be made based on an address in the request and a lookup table that lists mirrored addresses, specific address ranges, or the like. At operation 914, the MPU may copy the write request to create a first write request and a second write request. At operation 916, the MPU may transmit the first write request to the first memory device through the memory fabric interface. For example, using a table, the MPU may determine the first memory device based on the memory address and set one or more routing fields of the request so that the request is routed to the first memory device. At operation 918, the MPU may transmit the second write request to the second memory device through the memory fabric interface. For example, using the table, the MPU may determine the second memory device based on the memory address and set one or more routing fields of the request so that the request is routed to the second memory device.

[0053] In some examples, the MPU may wait for a response from each memory device. If both devices successfully reply, the MPU may forward one or more of the reply messages to the host. If one or more of the devices times out, the MPU may try a different configuration or route for one or more devices. If the MPU receives a successful reply message from one or more devices, the successful reply may be forwarded to the host. If a failure occurs again on one or more devices, the MPU may send a failure indication, such as a failover indication, to the host. The failover indication may trigger one or more processes for replacing the failed device, such as by updating the read and / or write targets so that future read and / or write requests are directed to the memory device that is still working.

[0054] At operation 920, the MPU may receive a read request for a second memory location. At operation 922, the MPU may identify that the second memory location is a mirrored memory location corresponding to the pair of memory devices. For example, by referencing a memory location in a table of paired memory locations. At operation 924, the MPU may determine a master device that satisfies the read command. For example, based on one or more bits in the address, a polling scheme (where requests for a particular device pair are handled by alternating devices), determining whether the first device or the second device is currently busy (and therefore servicing a read request from a non-busy device), or the like. At operation 926, the read request may be transmitted to the master device.

[0055] If a read request sent to the primary device is unsuccessful, an alternate fabric or route may be tried. If an alternate fabric or route is unsuccessful, a secondary device may be tried. If a read request is unsuccessful when sent to a secondary device, an alternate fabric or route may be tried. If an alternate fabric or route is unsuccessful, an error may be sent to the host.

[0056] Fig.10 Figure 1000 shows the roles for removing and replacing non-functional devices according to some examples of the present disclosure. At operation 1005, the administrator of the system removes the non-functional device. At operation 1010, the new device is installed, and at operation 1015, the new device is configured. At this point, each tenant A to N unlocks its encryption key at operation 1020-A...1020-N. This unlocking can be done via one or more applications executed with the tenant's permission. At operation 1025, the write target is updated. At operation 1020-A...1030-N, each tenant reads, makes modifications, and refreshes entries in the mirrored memory. At operation 1035, the system then updates the read target.

[0057] Fig.11 A block diagram of an example machine 1100 on which any one or more of the techniques (e.g., methodology) discussed herein may be executed is shown. In alternative embodiments, the machine 1100 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine 1100 may operate as a server machine, a client machine, or both in a server-client network environment. In an example, the machine 1100 may act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. The machine 1100 may be in the form of a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a network appliance, a network router, a switch or a bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions taken by the machine. In addition, although only a single machine is described, the term "machine" should also be deemed to include any collection of machines that individually or jointly execute one (or more) instruction sets to execute any one or more of the methods discussed herein (e.g., cloud computing, software as a service (SaaS), other computer cluster configurations). The machine 1100 may be part of or implement a distributed memory architecture, such as Figure 1 and 7 Demonstrated; implemented Figure 8 Components; Figures 2 to 4 Message passing flows; and Figure 5 , 6 , 9 and 10 methods.

[0058] As described herein, an instance may include or may operate on one or more logical units, components, or mechanisms (hereinafter referred to as "components"). A component is a tangible entity (e.g., hardware) that is capable of performing a specified operation and may be configured or arranged in a certain manner. In an example, a circuit may be arranged in a specified manner (e.g., internally or relative to an external entity such as other circuits) as a component. In an example, all or part of one or more computer systems (e.g., independent, client or server computer systems) or one or more hardware processors may be configured by firmware or software (e.g., instructions, application portions, or applications) as a component that operates to perform a specified operation. In an example, the software may reside on a machine-readable medium. In an example, when executed by the underlying hardware of the component, the software causes the hardware to perform the specified operation of the component.

[0059] Thus, the term "component" is understood to include a tangible entity, an entity that is physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., temporarily) configured (e.g., programmed) to operate in a specific manner or to perform part or all of any operations described herein. In view of the instances of temporarily configured components, each component need not be instantiated at any one time. For example, where the component includes a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured into corresponding different components at different times. The software can configure the hardware processor accordingly, for example, to constitute a specific module at one time, and to constitute different components at different times.

[0060] The machine (e.g., computer system) 1100 may include one or more hardware processors, such as processor 1102. Processor 1102 may be a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof. The machine 1100 may include a main memory 1104 and a static memory 1106, some or all of which may communicate with each other via an interconnect (e.g., bus) 1108. An example of the main memory 1104 may include synchronous dynamic random access memory (SDRAM), such as double data rate memory, such as DDR4 or DDR5. The interconnect 1108 may be one or more interconnects of different types, such that one or more components may be connected using a first type of interconnect and one or more components may be connected using a second type of interconnect. Example interconnects may include a memory bus, a peripheral component interconnect (PCI), a peripheral component interconnect express (PCIe) bus, a universal serial bus (USB), or the like.

[0061] The machine 1100 may further include a display component 1110, an alphanumeric input device 1112 (e.g., a keyboard), and a user interface (UI) navigation device 1114 (e.g., a mouse). In an example, the display component 1110, the input device 1112, and the UI navigation device 1114 may be a touch screen display. The machine 1100 may additionally include a storage device (e.g., a drive unit) 1116, a signal generating device 1118 (e.g., a speaker), a network interface device 1120, and one or more sensors 1121, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. The machine 1100 may include an output controller 1128, such as a serial (e.g., universal serial bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection for communicating or controlling one or more peripheral devices (e.g., printers, card readers, etc.).

[0062] The storage device 1116 may include a machine-readable medium 1122 on which is stored one or more sets of data structures or instructions 1124 (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions 1124 may also reside, completely or at least partially, within the main memory 1104, within the static memory 1106, or within the hardware processor 1102 during execution thereof by the machine 1100. In an example, one or any combination of the hardware processor 1102, the main memory 1104, the static memory 1106, or the storage device 1116 may constitute a machine-readable medium.

[0063] Although the machine-readable medium 1122 is illustrated as a single medium, the term "machine-readable medium" may include a single medium or multiple media (such as a centralized or distributed database and / or associated caches and servers) configured to store one or more instructions 1124.

[0064] The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions executed by the machine 1100 and causing the machine 1100 to perform any one or more of the techniques of the present disclosure or capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting machine-readable media examples may include solid-state memory and optical and magnetic media. Specific examples of machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable magnetic disks; magneto-optical disks; random access memory (RAM); solid-state drives (SSD); and CD-ROM and DVD-ROM disks. In some examples, machine-readable media may include non-transitory machine-readable media. In some examples, machine-readable media may include machine-readable media that is not a transient propagating signal.

[0065] The instructions 1124 may be further transmitted or received via the network interface device 1120 using a transmission medium over a communication network 1126. The machine 1100 may communicate with one or more other machines by wire or wireless communication using any of a number of transmission protocols (e.g., frame relay, Internet Protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile phone network (e.g., a cellular network), a plain old telephone (POTS) network, and a wireless data network (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards (known as ), IEEE 802.15.4 series of standards, 5G new air interface (NR) series of standards, long term evolution (LTE) series of standards, universal mobile telecommunication system (UMTS) series of standards, peer-to-peer (P2P) network, etc.). In an example, the network interface device 1120 may include one or more physical jacks (e.g., Ethernet, coaxial or telephone jacks) or one or more antennas to connect to the communication network 1126. In an example, the network interface device 1120 may include multiple antennas to use at least one of single input multiple output (SIMO), multiple input multiple output (MIMO), or multiple input single output (MISO) technology for wireless communication. In some examples, the network interface device 1120 may use multi-user MIMO technology for wireless communication.

[0066] Example 1 is a method for fault tolerance in a distributed memory system, the method comprising: at a memory processing unit connected to a host across a host interface and connected to multiple memory devices across a memory fabric interface: receiving a write request to a first memory location from the host across the host interface; identifying that the first memory location is a mirrored memory location corresponding to a pair of memory devices including a first memory device and a second memory device; copying the write request to create a first write request and a second write request; transmitting the first write request to the first memory device through the memory fabric interface; and transmitting the second write request to the second memory device through the memory fabric interface.

[0067] In Example 2, the subject matter of Example 1 includes: receiving a read request for a second memory location; identifying that the second memory location is a mirrored memory location corresponding to the pair of memory devices; determining a primary memory device as the first memory device or the second memory device; and transmitting the read request to the primary memory device through the memory fabric interface.

[0068] In Example 3, the subject matter of Example 2 includes, wherein determining the main memory device as the first memory device or the second memory device comprises utilizing one or more bits in an address of the second memory location.

[0069] In Example 4, the subject matter of Examples 2-3 includes: setting a timer before transmitting the read request to the main memory device; receiving expiration of the timer before receiving a response from the main memory device; and in response to receiving expiration of the timer before receiving the response from the main memory device, retransmitting the read request to the main memory device over an alternative memory fabric link.

[0070] In Example 5, the subject matter of Example 4 includes: setting a second timer before retransmitting the read request to the primary memory device over the alternate memory fabric link; receiving expiration of the second timer before receiving the response from the primary memory device over the alternate memory fabric link; and in response to receiving expiration of the second timer before receiving the response from the primary memory device over the alternate memory fabric link, retransmitting the read request to the secondary memory device.

[0071] In Example 6, the subject matter of Examples 1 to 5 includes: determining that the first memory device is to be replaced; in response to determining that the first memory device is to be replaced: updating a read target data structure to identify that the first memory device is out of service; updating a write target data structure to identify that the first memory device is out of service; and after updating the read target data structure and the write target data structure, using only the second memory device to service subsequent read or write requests for addresses corresponding to the pair of memory devices by sending the read or write requests only to the second memory device.

[0072] In Example 7, the subject matter of Example 6 includes: identifying a replacement memory device that has been installed for the first memory device; updating the write target data structure to identify that the replacement memory device and the second memory device are a pair of memory devices; reading data from the second memory device; setting the data from the second memory device to be modified; refreshing the data from the second memory device; and updating the read target data structure to identify that the replacement memory device and the second memory device are a pair of memory devices.

[0073] Example 8 is a memory processing unit for providing fault tolerance in a distributed memory system, the memory processing unit comprising: a processor configured to perform operations, the operations comprising: receiving a write request to a first memory location from a host across a host interface; identifying that the first memory location is a mirrored memory location corresponding to a pair of memory devices including a first memory device and a second memory device; copying the write request to create a first write request and a second write request; transmitting the first write request to the first memory device through a memory fabric interface; and transmitting the second write request to the second memory device through the memory fabric interface.

[0074] In Example 9, the subject matter of Example 8 includes, wherein the operations further include: receiving a read request for a second memory location; identifying that the second memory location is a mirrored memory location corresponding to the pair of memory devices; determining a primary memory device as the first memory device or the second memory device; and transmitting the read request to the primary memory device through the memory fabric interface.

[0075] In Example 10, the subject matter of Example 9 includes where determining the main memory device as the first memory device or the second memory device comprises utilizing one or more bits in an address of the second memory location.

[0076] In Example 11, the subject matter of Examples 9-10 includes, wherein the operations further include: setting a timer before the read request is transmitted to the main memory device; receiving an expiration of the timer before receiving a response from the main memory device; and in response to receiving the expiration of the timer before receiving the response from the main memory device, retransmitting the read request to the main memory device over an alternative memory fabric link.

[0077] In Example 12, the subject matter of Example 11 includes, wherein the operations further include: setting a second timer before retransmitting the read request to the primary memory device over the alternate memory fabric link; receiving an expiration of the second timer before receiving the response from the primary memory device over the alternate memory fabric link; and in response to receiving an expiration of the second timer before receiving the response from the primary memory device over the alternate memory fabric link, retransmitting the read request to the secondary memory device.

[0078] In Example 13, the subject matter of Examples 8 to 12 includes, wherein the operations further include: determining that the first memory device is to be replaced; in response to determining that the first memory device is to be replaced: updating a read target data structure to identify that the first memory device is out of service; updating a write target data structure to identify that the first memory device is out of service; and after updating the read target data structure and the write target data structure, using only the second memory device to service subsequent read or write requests for addresses corresponding to the pair of memory devices by sending the read or write requests only to the second memory device.

[0079] In Example 14, the subject matter of Example 13 includes, wherein the operation further includes: identifying a replacement memory device that has been installed for the first memory device; updating the write target data structure to identify that the replacement memory device and the second memory device are a pair of memory devices; reading the data from the second memory device; setting the data from the second memory device to be modified; refreshing the data from the second memory device; and updating the read target data structure to identify that the replacement memory device and the second memory device are a pair of memory devices.

[0080] Example 15 is a non-transitory machine-readable medium of storage instructions for fault tolerance in a distributed memory system, which instructions, when executed by a memory processing unit connected to a host across a host interface and to multiple memory devices across a memory fabric interface, cause the memory processing unit to perform operations, the operations including: receiving a write request to a first memory location from the host across the host interface; identifying that the first memory location is a mirrored memory location corresponding to a pair of memory devices including a first memory device and a second memory device; copying the write request to create a first write request and a second write request; transmitting the first write request to the first memory device through the memory fabric interface; and transmitting the second write request to the second memory device through the memory fabric interface.

[0081] In Example 16, the subject matter of Example 15 includes, wherein the operations further include: receiving a read request for a second memory location; identifying that the second memory location is a mirrored memory location corresponding to the pair of memory devices; determining a primary memory device as the first memory device or the second memory device; and transmitting the read request to the primary memory device through the memory fabric interface.

[0082] In Example 17, the subject matter of Example 16 includes, wherein determining the main memory device as the first memory device or the second memory device comprises utilizing one or more bits in an address of the second memory location.

[0083] In Example 18, the subject matter of Examples 16-17 includes, wherein the operations further include: setting a timer before transmitting the read request to the main memory device; receiving an expiration of the timer before receiving a response from the main memory device; and in response to receiving the expiration of the timer before receiving the response from the main memory device, retransmitting the read request to the main memory device over an alternative memory fabric link.

[0084] In Example 19, the subject matter of Example 18 includes, wherein the operations further include: setting a second timer before retransmitting the read request to the primary memory device over the alternate memory fabric link; receiving an expiration of the second timer before receiving the response from the primary memory device over the alternate memory fabric link; and in response to receiving an expiration of the second timer before receiving the response from the primary memory device over the alternate memory fabric link, retransmitting the read request to the secondary memory device.

[0085] In Example 20, the subject matter of Examples 15 to 19 includes, wherein the operations further include: determining that the first memory device is to be replaced; in response to determining that the first memory device is to be replaced: updating a read target data structure to identify that the first memory device is out of service; updating a write target data structure to identify that the first memory device is out of service; and after updating the read target data structure and the write target data structure, using only the second memory device to service subsequent read or write requests for addresses corresponding to the pair of memory devices by sending the read or write requests only to the second memory device.

[0086] In Example 21, the subject matter of Example 20 includes, wherein the operation further includes: identifying a replacement memory device that has been installed for the first memory device; updating the write target data structure to identify that the replacement memory device and the second memory device are a pair of memory devices; reading data from the second memory device; setting the data from the second memory device to be modified; refreshing the data from the second memory device; and updating the read target data structure to identify that the replacement memory device and the second memory device are a pair of memory devices.

[0087] Example 22 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any of Examples 1-21.

[0088] Example 23 is an apparatus comprising means for implementing any one of examples 1-21.

[0089] Example 24 is a system for implementing any one of examples 1 to 21.

[0090] Example 25 is a method for implementing any one of examples 1 to 21.

Claims

1. A method for fault tolerance in a distributed memory system, the method comprising: At a memory processing unit connected to a host across a host interface and connected to a plurality of memory devices across a memory fabric interface: receiving a write request to a first memory location from the host across the host interface; identifying that the first memory location is a mirrored memory location corresponding to a pair of memory devices including a first memory device and a second memory device; duplicating the write request to create a first write request and a second write request; transmitting the first write request to the first memory device via the memory fabric interface; and The second write request is transmitted to the second memory device through the memory fabric interface.

2. The method according to claim 1, further comprising: receiving a read request for a second memory location; identifying that the second memory location is a mirrored memory location corresponding to the pair of memory devices; determining a primary memory device as the first memory device or the second memory device; and The read request is transmitted to the main memory device through the memory fabric interface.

3. The method of claim 2, wherein determining the main memory device as the first memory device or the second memory device comprises utilizing one or more bits in an address of the second memory location.

4. The method according to claim 2, further comprising: setting a timer before transmitting the read request to the main memory device; a receiving timer expiring before receiving a response from the primary memory device; and In response to receiving an expiration of the timer prior to receiving the response from the primary memory device, the read request is retransmitted to the primary memory device over an alternate memory fabric link.

5. The method according to claim 4, further comprising: setting a second timer before retransmitting the read request to the primary memory device via the alternate memory fabric link; receiving an expiration of a second timer prior to receiving the response from the primary storage device via the alternate storage fabric link; and In response to receiving an expiration of the second timer prior to receiving the response from the primary memory device over the alternate memory fabric link, retransmitting the read request to a secondary memory device.

6. The method according to claim 1, further comprising: determining that the first memory device is to be replaced; In response to determining that the first memory device is to be replaced: updating a read target data structure to identify that the first memory device is out of service; updating a write target data structure to identify that the first memory device is out of service; and After updating the read target data structure and the write target data structure, subsequent read or write requests for addresses corresponding to the pair of memory devices are serviced using only the second memory device by sending read or write requests only to the second memory device.

7. The method according to claim 6, further comprising: identifying a replacement memory device installed for the first memory device; updating the write target data structure to identify that the replacement memory device and the second memory device are a pair of memory devices; reading data from the second memory device; setting the data from the second memory device as modified; refreshing the data from the second memory device; and The read target data structure is updated to identify that the replacement memory device and the second memory device are a pair of memory devices.

8. A memory processing unit for providing fault tolerance in a distributed memory system, the memory processing unit comprising: A processor configured to perform operations comprising: receiving a write request to a first memory location from a host across a host interface; identifying that the first memory location is a mirrored memory location corresponding to a pair of memory devices including a first memory device and a second memory device; duplicating the write request to create a first write request and a second write request; transmitting the first write request to the first memory device through a memory fabric interface; and The second write request is transmitted to the second memory device through the memory fabric interface.

9. The memory processing unit of claim 8, wherein the operations further comprise: receiving a read request for a second memory location; identifying that the second memory location is a mirrored memory location corresponding to the pair of memory devices; determining a primary memory device as the first memory device or the second memory device; and The read request is transmitted to the main memory device through the memory fabric interface.

10. The memory processing unit of claim 9, wherein the operation of determining the main memory device as the first memory device or the second memory device comprises utilizing one or more bits in an address of the second memory location.

11. The memory processing unit of claim 9, wherein the operations further comprise: setting a timer before transmitting the read request to the main memory device; a receive timer expiring before receiving a response from the primary memory device; and In response to receiving an expiration of the timer prior to receiving the response from the primary memory device, the read request is retransmitted to the primary memory device over an alternate memory fabric link.

12. The memory processing unit of claim 11, wherein the operations further comprise: setting a second timer before retransmitting the read request to the primary memory device via the alternate memory fabric link; receiving an expiration of a second timer prior to receiving the response from the primary storage device via the alternate storage fabric link; and In response to receiving an expiration of the second timer prior to receiving the response from the primary memory device over the alternate memory fabric link, retransmitting the read request to the secondary memory device.

13. The memory processing unit of claim 8, wherein the operations further comprise: determining that the first memory device is to be replaced; In response to determining that the first memory device is to be replaced: updating a read target data structure to identify that the first memory device is out of service; updating a write target data structure to identify that the first memory device is out of service; and After updating the read target data structure and the write target data structure, subsequent read or write requests for addresses corresponding to the pair of memory devices are serviced using only the second memory device by sending read or write requests only to the second memory device.

14. The memory processing unit of claim 13, wherein the operations further comprise: identifying a replacement memory device installed for the first memory device; updating the write target data structure to identify that the replacement memory device and the second memory device are a pair of memory devices; reading data from the second memory device; setting the data from the second memory device as modified; refreshing the data from the second memory device; and The read target data structure is updated to identify that the replacement memory device and the second memory device are a pair of memory devices.

15. A non-transitory machine-readable medium storing instructions for fault tolerance in a distributed memory system, the instructions, when executed by a memory processing unit connected to a host across a host interface and to a plurality of memory devices across a memory fabric interface, causing the memory processing unit to perform operations comprising: receiving a write request to a first memory location from the host across the host interface; identifying that the first memory location is a mirrored memory location corresponding to a pair of memory devices including a first memory device and a second memory device; duplicating the write request to create a first write request and a second write request; transmitting the first write request to the first memory device via the memory fabric interface; and The second write request is transmitted to the second memory device through the memory fabric interface.

16. The non-transitory machine-readable medium of claim 15, wherein the operations further comprise: receiving a read request for a second memory location; identifying that the second memory location is a mirrored memory location corresponding to the pair of memory devices; determining a primary memory device as the first memory device or the second memory device; and The read request is transmitted to the main memory device through the memory fabric interface.

17. The non-transitory machine-readable medium of claim 16, wherein the operation of determining the main memory device as the first memory device or the second memory device comprises utilizing one or more bits in an address of the second memory location.

18. The non-transitory machine-readable medium of claim 16, wherein the operations further comprise: setting a timer before transmitting the read request to the main memory device; a receiving timer expiring before receiving a response from the primary memory device; and In response to receiving an expiration of the timer prior to receiving the response from the primary memory device, the read request is retransmitted to the primary memory device over an alternate memory fabric link.

19. The non-transitory machine-readable medium of claim 18, wherein the operations further comprise: setting a second timer before retransmitting the read request to the primary memory device via the alternate memory fabric link; receiving an expiration of a second timer prior to receiving the response from the primary storage device via the alternate storage fabric link; and In response to receiving an expiration of the second timer prior to receiving the response from the primary memory device over the alternate memory fabric link, retransmitting the read request to the secondary memory device.

20. The non-transitory machine-readable medium of claim 15, wherein the operations further comprise: determining that the first memory device is to be replaced; In response to determining that the first memory device is to be replaced: updating a read target data structure to identify that the first memory device is out of service; updating a write target data structure to identify that the first memory device is out of service; and After updating the read target data structure and the write target data structure, subsequent read or write requests for addresses corresponding to the pair of memory devices are serviced using only the second memory device by sending read or write requests only to the second memory device.

Images