Chassis repair and migration in a vertically scaled numa system

Abstract

This application relates to chassis repair and migration in a scale-out NUMA system. One aspect of the application can provide a system and method to replace a failed node with a spare node in a non-uniform memory access (NUMA) system. During operation, in response to determining that a node migration condition is satisfied, the system can initialize a node controller of the spare node such that access to memory local to the spare node is to be handled by the node controller, stall the failed node and the spare node to allow state information of a processor on the failed node to be migrated to a processor on the spare node, and after unstalling the failed node and the spare node, migrate data from the failed node to the spare node while maintaining cache coherency in the NUMA system and the NUMA system remains operational, thereby facilitating continued execution of processes previously executing on the failed node.

Description

Background Technology

[0001] This disclosure generally relates to processor state and data migration when replacing a non-uniform memory access (NUMA) node that is about to fail. Attached Figure Description

[0002] Figure 1 An exemplary non-uniform memory access (NUMA) system is shown according to one aspect.

[0003] Figure 2 A flowchart illustrating an exemplary system initialization process according to one aspect is shown.

[0004] Figure 3A A flowchart illustrating an exemplary process for activating a standby node according to one aspect is presented.

[0005] Figure 3B A flowchart illustrating an exemplary process for migrating processor states according to one aspect is presented.

[0006] Figure 3C A flowchart illustrating an exemplary process for data migration in a service, based on one aspect, is presented.

[0007] Figure 3D A flowchart illustrating an exemplary process for data migration in a service, based on one aspect, is presented.

[0008] Figure 3E A flowchart illustrating an exemplary process for data migration in a service, based on one aspect, is presented.

[0009] Figure 4A A flowchart illustrating an exemplary process for data migration in a service, based on one aspect, is presented.

[0010] Figure 4B A flowchart illustrating an exemplary process for data migration in a service, based on one aspect, is presented.

[0011] Figure 4C A flowchart illustrating an exemplary process for data migration in a service, based on one aspect, is presented.

[0012] Figure 5A A flowchart illustrating an exemplary process for activating a standby node according to one aspect is presented.

[0013] Figure 5B A flowchart illustrating an exemplary process for data migration in a service, based on one aspect, is presented.

[0014] Figure 5CA flowchart illustrating an exemplary process for data migration in a service, based on one aspect, is presented.

[0015] Figure 5D A flowchart illustrating an exemplary process for data migration in a service, based on one aspect, is presented.

[0016] Figure 6 A block diagram of an exemplary node controller based on one aspect is presented.

[0017] Figure 7 An exemplary partitioned NUMA system based on one aspect is shown.

[0018] Figure 8 An exemplary computer system is shown that facilitates node migration according to one aspect.

[0019] In the accompanying drawings, the same reference numerals refer to the same elements. Detailed Implementation

[0020] The following description is presented to enable any person skilled in the art to make and use the examples, and is provided in the context of a particular application and its requirements. Those skilled in the art will readily understand various modifications to the disclosed examples, and the general principles defined herein may be applied to other examples and applications without departing from the spirit and scope of the invention. Therefore, the scope of this disclosure is not limited to the examples shown, but is consistent with the widest scope consistent with the principles and features disclosed herein.

[0021] As computer systems grow in size, the likelihood of component failures also increases. Failed components negatively impact the availability of a computer system. Some types of component failure (e.g., a faulty memory chip) may only result in performance degradation, while others (e.g., a faulty processor) can lead to system-wide failure. Various mechanisms can be used to monitor the state of a computer system, and analytics can be used to predict impending component failures. The goal is to prevent performance degradation or system failure by replacing potentially failing components with redundant ones while the system is running, before the actual failure occurs.

[0022] Modern operating systems (such as Linux) can add or remove memory modules while the system is running by providing spare memory modules. When needed (e.g., when a memory failure is anticipated), the spare memory module can be activated to replace the failed memory module. With the help of memory virtualization, data stored in the failed memory can be migrated to the newly activated memory. For example, data can be read from the physical address of the failed memory and then written to the corresponding physical address in the newly activated memory while updating the virtual address mapping. However, this solution only addresses failures in memory modules or memory controllers and cannot resolve non-memory failures, such as node failures in a computer cluster.

[0023] In Non-Unified Memory Access (NUMA) systems, clusters of microprocessors in a multiprocessor system can share resources such as memory, thereby improving system performance and expanding system capabilities. Each node in a NUMA system contains a processor and memory. An advanced memory controller (also known as a node controller) allows nodes to use memory on all other nodes. Data can be transferred via a NUMA connection when a processor accesses memory that is not within its own node (also known as remote memory).

[0024] Managing hardware failures in a NUMA system differs from managing failures in a computer cluster that does not implement NUMA. More specifically, a computer cluster can manage failures in a single node by shutting down the failed node and redistributing the failed job to the remaining nodes. Because each node is a fully contained computer system running its own copy of the operating system, process migration is unnecessary. Instead, jobs and processes can be restarted from the beginning or from an intermediate checkpoint. In contrast, in a NUMA system, multiple processors operate under a single operating system and access each other's memory. Maintaining cache consistency of data stored across different nodes can be challenging when a node destined for failure is replaced by a standby node.

[0025] Please note that some hypervisor-based solutions can combine small, independent computers into larger virtual NUMA systems. These solutions allow for the addition and removal of nodes to properly configure the system for computational tasks. However, in a virtual NUMA system, resources (both processors and memory) are allocated to the virtual machine. Resources are not added or removed to dynamically resolve impending hardware failures. A solution is desired to manage arbitrary hardware failures within a NUMA system, especially in vertically scaled NUMA systems where each node comprises a large number of processors and correspondingly larger amounts of memory.

[0026] According to some aspects of this application, a vertically scaled NUMA system may include a backup or standby node that can be activated when an impending failure is detected on a running node. While both nodes are running, the processor state and data of the failed node can be migrated to the standby node. More specifically, the migration can be an "in-place" migration, meaning the standby node becomes an exact replacement for the failed node in the address mapping, making the migration transparent to the operating system. The migration can be hardware-assisted, with migration hardware features added to each node controller. Depending on the cache coherency tracking scheme implemented by the node controller and processor, the migration process can be handled differently to maintain cache coherency. If the processor and node controller have a complete memory directory, a new "ForwardPending" directory state is introduced to indicate whether the migration of each cache line is complete. Note that a cache line is a unit of data transferred between memory and cache, also referred to as a cache block. Cache entries are created when a cache line is copied from memory to the cache. If the processor and node controller have a partial directory, different mechanisms can be used to prevent cache lines in the standby node's local memory from being accessed before the migration. For example, the local memory of a standby node might become poisoned, leading to anomalies. Alternatively, processors on the standby node can be aware of ongoing migrations and can default to the node controller as the owner of any cache lines not tracked in the directory. Crucially, any requests to the standby node's local memory will first be managed by the node controller, which tracks the migration status of each cache line. Using the proposed in-place migration, the entire system is only briefly paused to migrate processor states, and the data migration does not affect system operation. System downtime is significantly shorter than that required to reboot the system.

[0027] Figure 1 An exemplary NUMA system according to one aspect is presented. NUMA system 100 may include multiple active nodes (e.g., nodes 102 and 104) and at least one standby node (e.g., node 106). During normal operation, active nodes may perform computational tasks (e.g., run applications), while standby node 106 may be in standby mode (e.g., it may be partially initialized or powered off). The structure of each node may be similar, with each node including a node controller and multiple processing units (or processors) coupled to the node controller. Different processors and node controllers may also have similar hardware / software characteristics and be capable of performing similar functions. For example, active node 102 may include node controller 108 and processing units 110-116. In this disclosure, the terms "processing unit" and "processor" are interchangeable. Furthermore, because processors are connected to printed circuit boards (PCBs) via slots, a multiprocessor system may also be referred to as a multi-slot system.

[0028] The processing units and node controller within each node can be coupled to each other via intra-node links (e.g., links 118 and 120). Intra-node links between two processing units allow one processing unit to access memory attached to the other processing unit. Figure 1 (Not shown in the diagram). For a given processing unit, its own attached memory and the memory of other processing units attached to the same node are considered local memory. Different node controllers can be coupled to each other via inter-node links (or NUMA links) (e.g., links 122, 124, and 126). As previously described, processing units can access remote memory (i.e., the memory of processing units attached to other nodes) via inter-node links. Figure 1 In the diagram, links 124 and 126 between nodes are represented by dashed lines, indicating that these links are currently inactive because backup node 106 is currently inactive.

[0029] According to some aspects, when an impending failure is detected in one of the active nodes, a standby node 106 can be activated, and previously inactive inter-node links can also be activated. For example, if an impending failure is detected on node 102, inter-node links 124 and 126 will be activated. The activated inter-node link 126 allows processor state and data stored in the memory of node 102 to be migrated from the failed node 102 to the standby node 106, making the standby node 106 an exact replacement for the failed node 102. Once the migration of processor state and data is complete, the failed node 102 and its coupled inter-node links can be powered down. To reduce system downtime, both the failed node 102 and the standby node 106 can be running during the data migration. Therefore, processors on both nodes may access and modify the same data. Furthermore, when data is migrated from the failed node 102 to the standby node 106, other active nodes in the system (e.g., node 104) can also access and modify the same data. Maintaining data consistency is crucial to ensuring the proper functioning of the system during and after the data migration. Depending on some aspects, each node controller may include data migration logic, which can interact with other logical units on the node controller (e.g., directory logic or other types of logic involved in node migration operations) to ensure data consistency during node migration. Figure 1 In the example shown, node controller 108 may include data migration logic 128.

[0030] The following discussion on the data migration process can be found here. Figure 1 The system diagram shown. Note that, although... Figure 1A 12-slot, 3-node controller system is shown, but in reality, a NUMA system can include any reasonable number of nodes, and each node can include any reasonable number of processing units. According to some aspects, each node (including the node controller and processing units) can be packaged in a rack, and Figure 1 The system shown can also be referred to as a three-rack system. The action of replacing or repairing a faulty node can also be referred to as replacing or repairing a faulty chassis.

[0031] Before being put into service, the NUMA system (including active nodes and optional standby nodes) needs to be initialized. Figure 2 A flowchart illustrating an exemplary system initialization process according to one aspect is shown. During initialization, the system boot firmware can discover all processing units (or processors) and node controllers, including processors and node controllers on standby nodes (operation 202). The system boot firmware can initialize the node controller of the active node (also referred to as the active node controller) to create a NUMA system with a corresponding number of processors (operation 204). Figure 1 The system shown is an example where the system boot firmware can initialize the node controllers on nodes 102 and 104 to create an 8-processor or 8-socket NUMA system. Note that initializing the node controllers may include setting up address tables (which indicate the memory range for each processor) and routing tables for remote memory access.

[0032] The system boot firmware can also initialize the standby node's node controller (referred to as the standby node controller) (operation 206). Various parameters of this standby node controller can be configured to match an 8-processor system consisting of two active nodes. The system boot firmware can then boot the operating system from the processors on the active nodes (operation 208). Figure 1 The system shown is an example where the system boot firmware can boot the operating system of an 8-processor system including nodes 102 and 104. After booting, the NUMA system can start and run (Operation 210), and the standby node can remain partially initialized or powered down. Similarly, the inter-node links connected to the standby node can be powered down or disabled.

[0033] When the NUMA system is operating normally (i.e., no failures are expected), the standby node remains inactive (partially initialized or powered down). The hardware-based redirection table (which can be used to redirect remote memory accesses from the failed node to the standby node) is not initialized because it is unknown which node might fail. Similarly, on the standby node, the processor source address decoder (SAD) is currently unknown because it will vary depending on which active node the standby node will replace.

[0034] The initialization of standby nodes may differ slightly depending on the implementation of the cache coherence mechanism / protocol. In some cases, the system can implement a directory-based cache coherence protocol, where the directory is used to track the state of cache lines in local memory. In some cases, both the processor and the node controller have complete directories, allowing them to track the directory state of each cache line in local memory independently. In other cases, both the processor and the node controller have partial directories, tracking only the state of the cache lines they own. In the case of partial directories, only a subset of cache lines in local memory can have states other than "INVALID". If the partial directory approach is implemented, there are two solutions for maintaining cache coherence during data migration. If the first solution is chosen, all local memory may be poisoned during system initialization, meaning that memory is marked as bad or corrupted. The aim is to raise an exception when a cache line is referenced before proper migration. If the second solution is chosen, the processor is informed that a migration is occurring, and the node controller is assumed to be the owner of any cache lines not found in its directory.

[0035] Once the NUMA system detects that the active node is about to fail (but has not yet failed), the platform management system can be activated. Various known mechanisms can be used to detect impending failures, and the actual failure detection mechanism implemented by the system used is not limited to the scope of this disclosure. According to some aspects, the platform management system can be implemented as firmware embedded in the baseboard management controller (BMC) of the NUMA system. The platform management system can manage and control subsequent operations on the standby and active nodes. For example, once activated, the platform management system can enable the standby node, which may include powering on the standby node when needed, notifying the standby node of the identification of the failed node, completing the standby node initialization, etc.

[0036] Figure 3AA flowchart illustrating an exemplary process for activating a standby node according to one aspect is presented. In this example, each node's processor and node controller have full catalog knowledge of cache lines in local memory. During operation, the system detects the need for node migration and activates the platform management system (operation 302). As previously described, when an impending failure is detected on one of the active nodes, the processor state and data on the failed node can be migrated to the standby node, allowing the standby node to replace the failed node as an identical copy. The platform management system can then reboot the processor on the standby node (operation 304) and program / initialize the processor according to the failed node (operation 306). For example, the processor on the standby node can be programmed to have various parameters (e.g., SAD entries, Advanced Programmable Interrupt Controller (APIC) ID, etc.) that match (or are copies of) those on the failed node. This operation completes the initialization of the standby node. If the standby node was previously powered down, the platform management system can power on the standby node before initializing the processor on the standby node.

[0037] Subsequently, the standby node processor is paused (operation 308). Pausing the processor prevents execution on core or I / O traffic, thus preventing changes to data stored in local memory. The platform management system can notify the standby node controller (i.e., the node controller of the standby node) of the identification of the failed node (operation 310). In response to obtaining the identification of the failed node, the standby node controller can initialize a redirection table (operation 312). More specifically, the redirection table can indicate the memory ownership of the remaining active nodes (i.e., active nodes other than the failed node) and the standby node. The standby node controller can then request exclusive ownership of all local memory of the standby node (operation 314). For example, the standby node controller can issue "exclusive reads" to cache lines in local memory, one line at a time. Alternatively, some systems may include special features that allow setting processor directories for all cache lines at once. Granting the standby node controller exclusive ownership of cache lines ensures that local processors (i.e., processors on the standby node) send all their initial memory access requests to the standby node controller, thereby allowing the standby node controller to obtain the actual data from the failed node.

[0038] In a full catalog scenario, both the processor and the node controller are aware of all cache lines in the system and can add new catalog states to facilitate data migration. This new state, which can be called the "forwarding pending" state, can be represented by an additional bit in the catalog and can be used to indicate to the node controller that a cache line has not yet been migrated. Before migration, the standby node controller can initialize the catalog state of all local storage to the "forwarding pending" state (operation 316). The platform management system can then enable all inter-node links, including links between the standby node and all other nodes (operation 318). At this point, the standby node is fully activated and ready to receive processor state and data from the failed node.

[0039] Before data in memory can be migrated from a failed node to a standby node, the processor state should be migrated first. Figure 3B A flowchart illustrating an exemplary process for migrating processor state according to one aspect is presented. The entire system is paused (operation 320) before the processor state is migrated from the failed node to the standby node. Pausing the entire system, including the failed node, the remaining active nodes, and the standby node, prevents the processor state from becoming inconsistent during the migration. Once the system is paused, the failed node flushes its cache (operation 322). In other words, the failed node can write the contents of its cache back to its corresponding memory location. Because the failed node may be caching modified data from other active nodes, flushing the failed node's cache prevents future cache snooping from being sent to the failed node; instead, all future cache snooping will be redirected to the standby node. Subsequently, the state of each processor on the failed node can be migrated (i.e., copied) to each corresponding processor on the standby node (operation 324). Migrating state from one processor to another allows the other processor to continue executing processes previously executed on the original processor without requiring a reboot of one or both processors. After the state transition, the failed node and the remaining active nodes can each update their Global Address Mapping (GAM) tables to indicate that the standby node is the new owner of the migrated memory (operation 326). Thus, when an active node writes a cache line previously cached for the failed node back to memory, that cache line will be written back to the standby node's memory, not the failed node's. The system is then de-suspended (operation 328).

[0040] from Figure 3B As can be seen, pausing the entire system allows for a transition in processor state. During a pause, the system is unavailable. However, this system downtime is much shorter than the time required to reboot the system. Figure 3B The system pause-unpause process shown typically takes a few microseconds, while rebooting the system may take at least a few minutes to complete.

[0041] The processor on the standby node is now an exact copy of the processor on the failed node and can continue executing processes that were running on the failed node before the system stalled. However, the cache and local memory on the standby node are currently empty, and data needs to be migrated from the failed node to the standby node. Because NUMA systems have shared memory and a single operating system, cache coherency needs to be maintained during data migration. According to some aspects, data can be migrated from the failed node to the standby node during normal operation of the NUMA system. In other words, memory data can be migrated from the failed node to the standby node while the standby node is executing a process (e.g., a process that was initially running on the failed node).

[0042] Figure 3C A flowchart illustrating an exemplary process for data migration in a service, according to one aspect, is presented. When the standby node processor executes a process referencing (e.g., reading or writing) a local cache line, the processor checks the directory status of the local cache line, indicating that the standby node controller is the sole owner of the cache line (operation 332). The standby node controller then snoops / reads the cache line. The standby node controller checks its directory to determine if the directory status is "forward pending" (operation 334). If so, the standby node controller can consistently retrieve (e.g., issue a read request) the corresponding data from the failed node (operation 336) and write that data to the standby node's local memory (operation 338). Note that when the standby node controller retrieves data from the failed node, if a different active node happens to own the corresponding cache line (e.g., the data in the failed node's memory is outdated), the caches of the different active nodes will be snooped / read, and a response will be sent back to the failed node, which in turn sends the response to the standby node.

[0043] Once data is received, the directory status of the corresponding cache line in the standby node controller changes from "pending forwarding" to "invalid" (operation 340). Changing the "pending forwarding" status to "invalid" returns control of the cache line to the processor, which can later fetch the content from local memory into the cache. Future requests for the same memory address will now be managed only by the standby node (operation 342). At this point, the failed node is no longer in a loop for that memory address because the data at that memory address has been properly migrated to the standby node. If, in operation 334, the directory status is not "pending forwarding," the standby node controller will manage the request as a normal request (operation 342).

[0044] Figure 3C The process shown illustrates a scenario where data migration responds to a process executed by the processor being migrated. Data migration can also be triggered by different active nodes on a cache line requesting migration. Figure 3DA flowchart illustrating another exemplary process for data migration in a service, according to one aspect, is presented. During operation, different active nodes may request a cache line to be migrated (i.e., a cache line in a failed node) (Operation 350). The node controller of the active node checks its updated GAM table and sends the request to the standby node controller (Operation 352). The standby node controller checks its directory to determine if the directory status of the cache line is "forwarding pending" (Operation 354). If so, the standby node controller may consistently retrieve the corresponding data from the failed node (Operation 356), write the data to the standby node's local storage, and return the data to the requesting node (Operation 358). As in the previous example, if the corresponding cache line happens to be owned by different active nodes, the caches of the different active nodes will be snooped / read, and a response will be sent back to the failed node, which in turn sends the response to the standby node controller.

[0045] Once data is received, the directory status of the corresponding cache line in the standby node controller changes from "pending forwarding" to indicate that the requesting active node is the owner of that cache line (Operation 360). Future requests for the same memory address will now be managed only by the standby node (Operation 362). At this point, the failed node is no longer in the loop for that memory address because the data at that memory address has been properly migrated. If, in Operation 354, the directory status is not "pending forwarding," the standby node controller will manage the request as a normal request (Operation 362).

[0046] exist Figure 3C-3D In the example shown, data migration is performed on demand, meaning that data is migrated from the failed node to the standby node when either a local processor (i.e., the processor on the standby node) or a remote processor (i.e., the processor on another active node) requests data. In practice, another process can be... Figure 3C-3D The procedure shown operates concurrently to migrate memory data from the failed node to the standby node, even if the data has not yet been requested. Note that this procedure does not interfere with the normal operation of the NUMA system.

[0047] Figure 3EA flowchart illustrating an exemplary process for data migration in service, according to one aspect, is presented. During operation, the block transfer engine residing on the standby node controller can step through all cache lines in local memory by selecting a cache line (operation 370) and determining whether the selected cache line has a "forward pending" directory state (operation 372). The block transfer engine can step through memory addresses according to a predetermined order (e.g., ascending or descending). If the directory state is not "forward pending," the block transfer engine can skip the cache line (operation 374). If it is, the standby node controller can consistently retrieve the corresponding data from the failed node and write the data to the standby node's local memory (operation 376). Once the data is received, the directory state of the corresponding cache line in the standby node controller changes from "forward pending" to "invalid" (operation 378). At this point, the failed node is no longer in the loop for the specific address. Note that if the data in the address is being... Figure 3C If a memory address conflict occurs during the process migration shown in 3D, the block transfer engine can skip that address and move to the next address.

[0048] The block transfer engine can determine whether all buffer lines with a "forward pending" state have been properly migrated (operation 380). If not, it moves to the next buffer line (operation 370). If all buffer lines have been properly migrated (either via the block transfer engine or via...) Figure 3C and 3D If the process is as shown, the faulty node can be removed from all routing tables and powered off (operation 382). The faulty node can be replaced or repaired (operation 384), and the replaced or repaired node can be powered on and partially initialized as the next standby node (operation 386).

[0049] exist Figure 3C-3EIn the example shown, it is assumed that the processor and node controller on each node have a full directory of local memory. It is also possible that the processor and node controller only have a partial directory and cannot track the directory state for each cache line. In this case, introducing additional "forwarding pending" directory states is not feasible, and a different data migration solution may be required. In one solution, during the initialization of the standby node, the standby node's local memory can be marked as "poisoned," so that if a processor references a memory address before the cache line is properly migrated, an exception occurs. This exception can trigger an exception handler (which can be implemented as low-level firmware) residing on the remaining active node, which can then notify the standby node controller of the fault address, allowing the standby node controller to begin the correct data migration at the fault address. Note that if the standby node loses power while in standby, the local memory may be poisoned when the standby node is activated in response to an impending failure on one of the active nodes. More specifically, the local memory can be marked as poisoned when the processor on the standby node reboots. The path to the exception handler can be specified when the inter-node link between the standby node and other nodes is activated.

[0050] Without an additional "forwarding pending" directory status (which explicitly indicates the migration status of a cache line), the standby node controller may need to use different mechanisms to facilitate proper data migration. More specifically, the "poisoned" or "unpoisoned" status of a cache line can be used to indicate whether that cache line has been migrated. Figure 4A A flowchart illustrating an exemplary process for data migration in service, according to one aspect, is presented. This process occurs after the standby node has been activated and the processor state has been migrated from the failed node to the standby node. Note that during activation, the standby node controller no longer needs to request ownership of local storage, and there is no "forwarding pending" state in the directory. During operation, the standby node processor executes a thread referencing a local cache line (operation 402), and the processor determines whether that cache line contains poisoned data (operation 404). If so, the processor can suspend the thread and trigger an exception handler residing on the remaining active node (operation 406). The exception handler can then notify the standby node controller of the address of the cache line containing the poisoned data (operation 408). This essentially indicates to the standby node controller that the data in that cache line has not yet been copied from the failed node.

[0051] In response, the standby node controller can consistently retrieve the corresponding data from the failed node and write the retrieved data to local memory to replace the poisoned data (operation 410). Writing the data to local memory clears the poisoning state. If the corresponding cache line happens to be owned by different active nodes, the caches of the different active nodes will be snooped / read, and a response will be sent back to the failed node, which in turn sends the response to the standby node controller. Once the poisoned data has been replaced by good data, the exception handler will return control to the suspended thread, which can now use the correct data (operation 412). Future requests to the same memory address will now only be managed by the standby node as normal requests (operation 414). At this point, the failed node is no longer in a loop for that memory address because the data at that memory address has been properly migrated and is no longer poisoned. If, in operation 404, local memory is not poisoned, indicating that the data has been migrated, then the request will be managed as a normal request (operation 414).

[0052] Figure 4B A flowchart illustrating an exemplary process for data migration in a service, based on one aspect, is presented. During operation, different active nodes can request a cache line to be migrated (i.e., a cache line in a failed node) (Operation 420). The active node's node controller checks its updated GAM table and sends the request to the standby node controller (Operation 422). The standby node controller then sends the request to the corresponding local processor based on the memory address (Operation 424). The local processor determines whether the memory address contains poisoned data (Operation 426). If so, the local processor can respond to the request with a response indicating data poisoning (Operation 428). Once the poisoning response is received, the standby node controller can consistently retrieve the corresponding data from the failed node and change the directory state of the cache line to indicate that the requesting active node is the owner of the cache line (Operation 430). The standby node controller can write the data to local memory and clear the poisoning state (Operation 432). The standby node controller can return the data to the requesting active node (Operation 434). Future requests for the same memory address will now be managed only by the standby node (Operation 436). At this point, the faulty node is no longer in the loop for that memory address because the data at that memory address has been properly migrated. If, in operation 426, the local processor determines that the memory address is not poisoned, it returns the data to the standby node controller (operation 438). This non-poisoned response indicates to the standby node controller that the data has been migrated. In response, the standby node controller can return the data to the requesting active node (operation 434).

[0053] Similar to the full directory scenario above, in this partial directory scenario, the standby node controller can implement a block transfer engine to progressively traverse all local storage to facilitate data migration. Figure 4C A flowchart illustrating an exemplary process for data migration in service according to one aspect is presented. During operation, the block transfer engine residing on the standby node controller can progressively traverse all cache lines in local memory by selecting a cache line (operation 440) and determining whether the selected cache line has poisoned data (operation 442). More specifically, the block transfer engine can consistently read data from the selected cache line by sending a request to the corresponding processor, which responds to the request with a poisoned or non-poisoned response from the standby node controller. If the data is not poisoned, the block transfer engine skips the cache line (operation 444). If the data is poisoned, the standby node controller can consistently retrieve the corresponding data from the failed node and write the data to the standby node's local memory (operation 446). Once the data is received, the directory status of the corresponding cache line in the standby node controller is updated to "invalid" (operation 448). At this point, the failed node is no longer in the loop for this particular address. Note that if the data is being... Figure 4A If a conflict occurs at the same memory address due to process migration as shown in 4B, the block transfer engine can skip that address and move to the next address.

[0054] The block transfer engine can determine whether all cache lines have been properly migrated (operation 450). If not, it moves to the next cache line (operation 440). If all cache lines have been properly migrated (either via the block transfer engine or via...) Figure 4A and 4B If the process is as shown, the faulty node can be removed from all routing tables and powered off (operation 452). The faulty node can be replaced or repaired (operation 454), and the replaced or repaired node can be powered on and partially initialized as the next standby node (operation 456).

[0055] Poisoning the standby node's local memory during its initialization ensures that local memory access is always managed by the standby node controller, and the "poisoned" state can be used to track the migration status of each cache line. In different implementations, the processor can be aware of ongoing migrations and defaults to the standby node controller being the owner of any cache lines not found in the processor's directory, rather than poisoning local memory to force the standby node controller to control memory access. To track migration status, memory space can be allocated for the node controller to record the migration status of blocks / pages in local memory (e.g., by setting flags). Because this solution requires modification of the processor and allocation of memory space, the standby node activation process can differ from... Figure 3A The process is shown.

[0056] Figure 5A A flowchart illustrating an exemplary process for activating a standby node according to one aspect is presented. During operation, the system detects the need for node migration and activates the platform management system (operation 502). The platform management system can then reboot the processor on the standby node and complete the processor initialization (operation 504). For example, the processor on the standby node can be programmed to have various parameters matching those of the failed node (e.g., SAD entries, APIC ID, etc.). Note that if the standby node was previously powered down, the platform management system can power on the standby node before initializing the processor on the standby node.

[0057] Subsequently, the standby node processor is paused to prevent execution on core or I / O traffic (Operation 506). At this point, the processor's cache is empty, and the processor directory does not track any cache line owners. The processor can be configured to set the standby node controller as the default owner of any cache lines not tracked in the processor's directory (Operation 508). This effectively forces all initial local memory accesses to be managed by the standby node controller.

[0058] The platform management system can notify the standby node controller of the identifier of the failed node (operation 510). In response to receiving the identifier of the failed node, the standby node controller can initialize a redirection table to indicate the memory ownership of the remaining active and standby nodes (operation 512). The platform management system can then enable links between all nodes (operation 514).

[0059] The standby node controller is allocated unused memory space (Operation 516). Depending on some aspects, this allocated space may be attached to a local processor, a remote processor, or directly to the standby node controller. This allocated memory space can be used to store flags indicating whether cache lines or pages (depending on the implementation) have been migrated. A flag (which may be a binary bit) can be used for each cache line or page of the failed node. The standby node controller sets the default state of all local cache lines to "unmigrated" (Operation 518). Once the standby node is initialized, it uses a similar approach... Figure 3BThe process illustrated allows the state of a processor on a failed node to be migrated to that of a processor on a standby node. Once the processor state migration is complete, data migration from the failed node to the standby node can begin. Note that using a single flag to track the cache line migration state of a page (which may include multiple cache lines) reduces the number of flags required to track the migration state of all cache lines, in exchange for greater complexity. As cache lines within a page are migrated, the node controller can track all cache lines within the same page. The number of partially migrated pages can be capped without sacrificing tracking accuracy. Once a page has been successfully migrated (i.e., all cache lines in the page have been migrated), a per-page flag can be set, and the node controller can move to another page. Note that the page size cannot be too large, otherwise the processor on the standby node might generate an access pattern that forces the node controller to restrict requests. Depending on some aspects, each page can have two cache lines, which can halve the storage required for the "migrated" flag.

[0060] Figure 5B A flowchart illustrating an exemplary process for data migration in a service, according to one aspect, is presented. During operation, the standby node processor executes a process referencing the local cache line (operation 522). For example, the processor may issue a read or write request targeting the local cache line. Because local memory is not tracked in its directory, the processor sets a default directory state for the cache line, indicating that the standby node controller is the owner (operation 524). The standby node controller checks its directory to determine if a state can be found for the cache line (operation 526). If no state is found, the standby node controller accesses the allocated memory space to check the status flag (operation 528) and determines if the flag is set to "not migrated," indicating that data in the cache line has not been properly migrated (operation 530).

[0061] If the flag is set to "not migrated," the standby node controller can consistently retrieve the corresponding data from the failed node and write it to local memory (operation 532). If the corresponding cache line happens to be owned by a different active node, the caches of the different active nodes will be snooped / read, and a response will be sent back to the failed node, which in turn sends the response to the standby node controller. Note that the standby node controller does not need to change its directory state upon receiving data, as the data is returned to the processor. However, the standby node controller can update the migration state stored in the allocated memory space by clearing the flag indicating that the cache line / page has been migrated (operation 534). Future requests for the same memory address will now only be managed by the standby node as normal requests (operation 536). At this point, the failed node exits the memory address loop.

[0062] If the flag is cleared, indicating that a cache line / page has been migrated, the standby node controller will assume its directory state is "invalid" (Operation 538) and will process the memory transaction according to the implemented consensus protocol (Operation 540). If, in Operation 526, the standby node controller finds the state, it will process the memory transaction according to the implemented consensus protocol (Operation 540).

[0063] Figure 5C A flowchart illustrating another exemplary process for data migration in a service, based on one aspect, is presented. During operation, different active nodes can request the migration of cache lines (i.e., cache lines in failed nodes) (Operation 542). The node controller of the active node checks its updated GAM table and sends the request to the standby node controller (Operation 544). The remaining operations can be similar. Figure 5B The operation is shown below. The standby node controller determines whether the status of the cache line can be found in its directory (operation 546). If the status cannot be found, the standby node controller checks the migration status flag to determine whether the data in the cache line has been properly migrated (operation 548).

[0064] If the flag indicates an "unmigrated" state, the standby node controller can consistently retrieve the corresponding data from the failed node and write it to local memory (operation 550). If the corresponding cache line happens to be owned by a different active node, the caches of the different active nodes will be snooped / read, and a response will be sent back to the failed node, which in turn sends the response to the standby node controller. The standby node controller does not change its directory state but updates the migration status of the cache line / page to "migrated" (operation 552). Future requests for the same memory address will now be managed only by the standby node as normal requests (operation 554). At this point, the failed node exits the loop.

[0065] If the flag indicates a "migrated" state, the standby node controller will assume its directory state is "invalid" (operation 556) and will process the memory transaction according to the implemented consensus protocol (operation 558). If, in operation 546, the standby node controller finds the state, it will process the memory transaction according to the implemented consensus protocol (operation 558).

[0066] This solution for partial directory scenarios can also use a block transfer engine to gradually traverse all local storage to facilitate data migration. Figure 5DA flowchart illustrating an exemplary process for data migration in service according to one aspect is presented. During operation, the block transfer engine residing on the standby node controller can progressively traverse all cache lines in local memory by selecting a cache line (operation 560) and determining whether the selected cache line has a migration status flag set to "not migrated" (operation 562). If not, the cache line has been migrated, and the block transfer engine can skip the cache line (operation 564). If yes, the standby node controller can consistently retrieve the corresponding data from the failed node and write the data to the standby node's local memory (operation 566). Once the data is received, the migration status flag is updated to "migrated" (operation 568). At this point, the failed node is no longer in the loop for this particular address. Note that if due to Figure 5B If the process shown in 5C is migrating data and there is a conflict at the same memory address, the block transfer engine can skip that address and move to the next address.

[0067] The block transfer engine can determine whether all cache lines have been properly migrated (operation 570). If not, it moves to the next cache line (operation 560). If all cache lines have been properly migrated (either via the block transfer engine or via...) Figure 5B and 5C If the process is as shown in the diagram, the faulty node can be removed from all routing tables and powered off (operation 572). The faulty node can be replaced or repaired (operation 574), and the replaced or repaired node can be powered on and partially initialized as the next standby node (operation 576).

[0068] To facilitate the migration of processor state and data, additional hardware features can be added to existing node controllers. These additional hardware features can be provided by adding new hardware logic blocks (e.g., logic gates, registers, memory, etc.) or by modifying existing hardware logic blocks. Besides using hardware logic, software-assisted migration is also possible. These additional hardware features can also be included in the processor to allow migration in systems without node controllers. However, NUMA systems without node controllers are typically smaller, less prone to failure, and the cost of providing redundant additional equipment may be unreasonable. In the case of a single-processor system, migrating hardware may be an unnecessary burden because no spare processor is available.

[0069] Figure 6A block diagram of an exemplary node controller according to one aspect is presented. Node controller 600 may include initialization logic 602, redirection table logic 604, processor state transition logic 606, data transition logic 608, directory logic 610, request processing logic 612, transition status flag logic 614, block transfer engine 616, data read logic 618, data write logic 620, processor interface 622, and inter-node interface 624. Depending on the actual implementation solution, the number of components in node controller 600 may be greater than... Figure 6 More or less as shown. The various logic units within the node controller 600 can interact with each other to perform... Figure 2-5D The various operations shown.

[0070] When a standby node is activated, initialization logic 602 can be responsible for initializing various logic units in node controller 600. More specifically, by initializing various logic units, initialization logic 602 can ensure that access to the standby node's local memory is always handled by the node controller. Depending on the implemented solution, initialization logic 602 can set the directory state of cache lines in local memory, mark local memory as poisoned, or set a migration status flag (when available). Redirect table logic 604 implements a redirect table that indicates whether requests should be forwarded to remote memory. When a standby node is activated in response to an impending failure of the active node, initialization logic 602 can update the redirect table on the standby node to indicate that the memory of cache lines to be migrated belongs to the failed node.

[0071] Processor state transition logic 606 is responsible for transitioning the state of the processor on the failed node to the processor on the standby node. This may cause processor stalls throughout the NUMA system to prevent conflicts. Data transition logic 608 is responsible for transitioning data from the failed node to the standby node. Depending on the implementation solution, data transition logic 608 may interact with different logic units in the node controller 600 to facilitate correct data transitions.

[0072] Directory logic 610 can implement a full directory to track the directory state of the entire local memory, or a partial directory to track the directory state of a subset of cache lines in local memory. In a full directory implementation, in addition to traditional directory states (e.g., "modified," "owned," "exclusive," "shared," "invalid," etc.), directory logic 610 can also track new "forwarding pending" states, which indicate the migration status of the tracked cache line. Entries in the directory can be updated based on migration status. For example, after a cache line has been migrated from a failed node to a standby node, the "forwarding pending" state can be cleared (e.g., the state can be changed to "invalid" or indicate that the line is owned by the requesting node controller).

[0073] Request processing logic 612 can handle memory requests. Depending on the implementation, request processing logic 612 can perform different actions, such as checking the directory managed by directory logic 610, sending requests to the local processor, sending requests to other node controllers, etc. Migration status flag logic 614 can be responsible for setting and clearing migration flags (when implemented). Block transfer engine 616 can be responsible for step-by-step traversal of local memory to migrate unmigrated and unrequested (by local or remote processors) cache lines one line at a time.

[0074] Data read logic 618 is responsible for reading data from the memory of the failed node, and data write logic 620 is responsible for writing data to the local memory of the standby node. Processor interface 622 is the communication interface to the local processor, and inter-node interface 624 is the communication interface to other nodes.

[0075] In the example discussed above, the spare node is included in a single system with N active nodes to provide N+1 redundancy. In practice, these solutions can be extended to partitioned systems, where a large NUMA system is divided into multiple smaller partitions for reasons including isolating jobs from each other, or possibly adjusting the system size based on the computer jobs being run. In a partitioned system, a single redundant spare node can be shared between two or more partitions.

[0076] Figure 7 An exemplary partitioned NUMA system according to one aspect is illustrated. The partitioned NUMA system 700 may include partitions 702 and 704, each containing multiple NUMA nodes and a spare node 706. In this example, a 5-node, 20-processor system can be partitioned into two partitions, each containing two nodes, each node having four processors. In other words, each partition is an eight-processor system. Similar to... Figure 1 The nodes shown Figure 7 Each node in the system can include a node controller, and each node controller can include data migration logic for maintaining cache consistency during node migration. A standby node 706 can be on standby for two partitions, and the inter-node link between standby node 706 and other links is disabled. Whenever an impending failure is detected on any node, standby node 706 can be activated to replace the failed node. Processor state and data can be migrated using one of the methods described in the previous examples, depending on whether a full or partial directory is implemented, or whether the processor has been modified to be aware of the ongoing migration. Figure 7 In the example shown, two partitions share a standby node. Other configurations are also possible, such as three partitions sharing a standby node, or three partitions sharing two standby nodes.

[0077] Allowing multiple partitions to share a single redundant node, rather than one redundant node per partition, can offer cost-effectiveness benefits. The number of spare nodes provided in a computing system can be determined based on the overall failure probability of the computing system, rather than on the number of partitions within the computing system. This allows for a large number of spare nodes to be provided for systems with a high probability of failure.

[0078] Figure 8 An exemplary computer system facilitating node migration according to one aspect of this application is illustrated. The computer system 800 includes a processor 802, a memory 804, and a storage device 806. Furthermore, the computer system 800 may be coupled to a peripheral input / output (I / O) user device 810, such as a display device 812, a keyboard 814, and an indicating device 816. The storage device 806 may store an operating system 818, a node migration system 820, and data 850.

[0079] The node migration system 820 may include instructions that, when executed by the computer system 800, cause the computer system 800 or the processor 802 to perform the methods and / or processes described in this disclosure (e.g., Figure 2-5D (Various operations are shown in the diagram). Specifically, by executing these instructions, the computer system 800 can perform various functions to implement node migration while maintaining cache coherence. The node migration system 820 may include instructions for initializing the standby node controller (controller initialization instruction 822), instructions for implementing the redirection table (redirection table instruction 824), instructions for migrating the processor state (processor state migration instruction 826), instructions for migrating memory data (data migration instruction 828), instructions for implementing the directory (directory implementation instruction 830), instructions for processing memory requests (request processing instruction 832), instructions for implementing migration status flags (migration status flag instruction 834), instructions for implementing the block transfer engine (block transfer instruction 836), instructions for reading data from memory (data read instruction 838), instructions for writing data to memory (data write instruction 840), instructions for facilitating communication between the processor and the node controller (processor controller communication instruction 842), and instructions for facilitating communication between node controllers (inter-node communication instruction 844). Data 850 may include directory information 852.

[0080] The various solutions discussed above (including the full catalog solution and the two partial catalog solutions) each create a unique way to prevent failures by tracking the migration status of cache lines while replacing failed (but not yet failed) nodes with standby nodes, thus maintaining cache consistency during data migration. Maintaining a full catalog allows the standby node controller to track every possible cache line in local memory. More specifically, a newly created "forwarding pending" state can be used to track the migration status of each cache line. By granting the standby node controller exclusive ownership of local memory, all memory accesses are initially managed by the standby node controller, which relies on the "forwarding pending" state to determine the migration status of the cache line. In the case of partial catalogs, because the catalog does not track local memory, migration status needs to be tracked differently. In one solution, all local memory can initially be poisoned, forcing the standby node controller to manage data access. Once migrated, the data is no longer poisoned and can be accessed normally. In another solution, the processor can be modified to include a feature that sets the node controller as the owner of cache lines not tracked in its catalog. This also forces the node controller to manage all initial data accesses. A separate, unused memory space can be allocated to the node controller to include trace entries (flags) for cache lines / pages in local memory. Once a cache line is migrated, the flag can change from "not migrated" to "migrated". In addition to performing on-demand migration of cache lines (i.e., migrating a cache line when it is referenced), the system can also use a block transfer engine (which can run in the background) to progressively traverse memory to migrate cache lines that have not yet been requested.

[0081] One aspect of this application provides a system and method for replacing a failed node with a standby node in a Non-Unified Memory Access (NUMA) system. During operation, in response to determining that node migration conditions are met, the system can initialize the node controller of the standby node such that access to the local memory of the standby node will be handled by the node controller, pause both the failed node and the standby node to allow state information of the processor on the failed node to be migrated to the processor on the standby node, and after unpausing both the failed node and the standby node, migrate data from the failed node to the standby node while maintaining cache coherence in the NUMA system, and the NUMA system continues to operate, thereby facilitating the continuous execution of processes previously executed on the failed node.

[0082] In one variation of this approach, the node controller may maintain a full directory of the standby node's local memory. Initializing the node controller may include granting the node controller exclusive ownership of each cache line in the local memory and setting the directory state of each cache line in the local memory to a first state indicating that the data in the cache line has not yet been migrated from the failed node. Migrating data may include consistently retrieving cache lines from the failed node in response to determining that the directory state of the requested cache line is in the first state, and writing the retrieved cache lines to the standby node's local memory.

[0083] In another variant, cache lines are requested by the local processor of the standby node, and the system can update the directory status of the cache lines to a second state indicating that the cache lines are owned by the processor after writing the acquired cache lines.

[0084] In another variant, a node controller located far from the standby node requests a cache line, and after writing the acquired cache line, the system can update the cache line's directory status to a third state indicating that the cache line is owned by the remote node controller.

[0085] In one variation of this approach, the node controller may maintain a partial directory of local memory. Initializing the node controller may include marking each cache line in the local memory as corrupted. Migrating data may include consistently retrieving cache lines from the failed node in response to determining that a requested cache line in the standby node's local memory has been marked as corrupted, and writing the retrieved cache lines to the standby node's local memory.

[0086] In another variant, the local processor of the standby node requests a cache line, which triggers an exception and causes exception handlers on different nodes residing in the NUMA system to notify the node controller of the standby node of the address of the corrupted cache line.

[0087] In another variation, a node controller located away from the standby node requests a cache line, and the standby node controller can forward the request to the corresponding processor local to the standby node, which determines that the requested cache line is corrupt.

[0088] In one variation of this approach, the node controller may maintain a portion of the local memory's directory. Initializing the node controller may include allocating unused memory space to the node controller to allow it to track the migration status of each cache line in local memory and setting the migration status flag for each cache line to "not migrated".

[0089] In another variation, migration data may include consistently acquiring cache lines from the faulty node in response to determining that the migration status of a requested cache line in local memory is set to "not migrated", writing the acquired cache line to local memory, and setting the migration status of the written cache line to "migrated".

[0090] In one variation of this approach, migrating data from a failed node to a standby node may involve a block transfer engine iterating through local memory to migrate cache lines that have not yet been migrated and have not yet been requested.

[0091] One aspect of this application provides an apparatus for facilitating the replacement of a failed node with a standby node in a Non-Unified Memory Access (NUMA) system. The apparatus may include: node controller initialization logic for initializing the node controller of the standby node in response to determining that node migration conditions are met, such that access to local memory of the standby node will be handled by the node controller; processor state transition logic configured to pause both the failed node and the standby node to allow the state of the processor on the failed node to be transitioned to that of the processor on the standby node; and data transition logic for migrating data from the failed node to the standby node after unpausing both the failed node and the standby node, while maintaining cache coherence in the NUMA system and ensuring the NUMA system remains operational, thereby facilitating the continuous execution of processes previously executed on the failed node.

[0092] In a variation of this aspect, the apparatus may further include a directory for tracking the directory status of cache lines in local memory. This directory may include a full directory tracking each cache line in local memory, or a partial directory tracking a subset of cache lines in local memory.

[0093] The methods and processes described in the Detailed Description section may be embodied in code and / or data, which may be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system executes the methods and processes embodied in data structures and code and stored in the computer-readable storage medium.

[0094] Furthermore, the methods and processes described above may be incorporated into hardware modules or devices. These hardware modules or devices may include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors that execute specific software modules or pieces of code at specific times, and other programmable logic devices now known or developed in the future. When a hardware module or device is activated, it executes the methods and processes contained therein.

[0095] The foregoing description is for illustrative purposes only. It is not intended to be exhaustive or to limit the scope of this disclosure to the forms disclosed. Therefore, many modifications and variations will be apparent to those skilled in the art.

Claims

1. A method for replacing a failed node with a spare node in a non-uniform memory access (NUMA) system, the method comprising: In response to determining that the node migration conditions are met, the node controller of the standby node is initialized such that access to the local memory of the standby node will be handled by the node controller. Pause the faulty node and the backup node to allow the state information of the processor on the faulty node to be migrated to the processor on the backup node; After unblocking the faulty node and the standby node, data is migrated from the faulty node to the standby node while maintaining cache consistency in the NUMA system and keeping the NUMA system running, thereby facilitating the continuous execution of processes previously executed on the faulty node. A portion of the local memory directory is maintained at the node controller of the standby node; Initializing the node controller includes marking each cache line in the local memory as corrupted; and The data migration includes, in response to determining that a requested cache line in the local memory of the standby node is marked as damaged, consistently acquiring the cache line from the faulty node and writing the acquired cache line into the local memory of the standby node.

2. The method of claim 1, further comprising maintaining a full directory of the local memory of the standby node at the node controller of the standby node; in, Initializing the node controller includes granting the node controller exclusive ownership of each cache line in the local memory and setting the directory status of each cache line in the local memory to a first state, the first state indicating that the data in the cache line has not yet been migrated from the faulty node; as well as The data migration includes, in response to determining that the directory state of the requested cache line is the first state, consistently obtaining the cache line from the faulty node and writing the obtained cache line into the local memory of the standby node.

3. The method according to claim 2, wherein, The cache line is requested by the local processor of the standby node, and the method further includes: After writing to the acquired cache line, the directory status of the cache line is updated to a second status, which indicates that the cache line is owned by the processor.

4. The method according to claim 2, wherein, The cache line is requested by a node controller located remotely from the backup node, and the method further includes: After writing the acquired cache line, the directory status of the cache line is updated to a third state, which indicates that the cache line is owned by the remote node controller.

5. The method according to claim 1, wherein, The cache line is requested by the local processor of the standby node, which triggers an exception and causes exception handlers residing on different nodes in the NUMA system to notify the node controller of the standby node of the address of the corrupted cache line.

6. The method according to claim 1, wherein, The cache line is requested by a node controller located remotely from the backup node, and the method further includes: The node controller of the standby node forwards the request to the corresponding processor local to the standby node, and the corresponding processor determines that the requested cache line is corrupted.

7. The method according to claim 1, in, Initializing the node controller includes allocating unused memory space to the standby node, allowing the node controller to track the migration status of each cache line in the local memory, and setting the migration status of each cache line to "not migrated".

8. The method according to claim 7, wherein, The migration of the data includes: In response to determining that the migration status of the requested cache line in the local memory is set to "not migrated", the cache line is consistently obtained from the faulty node; Write the acquired cache line to the local memory; and Set the migration status of the written cache line to "migrated".

9. The method according to claim 1, wherein, Migrating data from the failed node to the standby node involves using a block transfer engine to progressively traverse the local memory to migrate cache lines that have not yet been migrated and have not yet been requested.

10. An apparatus for facilitating the replacement of a failed node with a spare node in a non-uniform memory access NUMA system, the apparatus comprising: The node controller initialization logic is used to initialize the node controller of the standby node in response to determining that the node migration conditions are met, so that access to the local memory of the standby node will be handled by the node controller. Processor state transition logic is used to pause the faulty node and the standby node to allow the state of the processor on the faulty node to be transitioned to the state of the processor on the standby node. as well as The data migration logic is used to migrate data from the faulty node to the standby node after the faulty node and the standby node are unblocked, while maintaining cache consistency in the NUMA system and keeping the NUMA system running, thereby facilitating the continuous execution of processes previously executed on the faulty node. A directory for tracking the directory status of cache lines in the local memory of the standby node; The directory includes a partial directory; The node controller initialization logic is used to mark each cache line in the local memory of the standby node as corrupted; and The data migration logic is configured to, in response to determining that a requested cache line in the local memory of the standby node is marked as damaged, consistently obtain the cache line from the faulty node and write the obtained cache line into the local memory of the standby node.

11. The apparatus according to claim 10, wherein, The directory includes the portion of the directory that tracks a subset of cache lines in the local memory.

12. The apparatus according to claim 11, in, The directory includes the full directory; The node controller initialization logic is used to grant the node controller of the standby node exclusive ownership of each cache line in the local memory of the standby node, and to set the directory status of each cache line to a first state, the first state indicating that the data in the cache line has not yet been migrated from the failed node; and The data migration logic is configured to consistently obtain the cache line from the faulty node and write the obtained cache line into the local memory of the standby node in response to determining that the directory state of the requested cache line is the first state.

13. The apparatus according to claim 12, wherein, The cache line is requested by the local processor of the standby node, and the data migration logic is used for: After writing to the acquired cache line, the directory status of the cache line is updated to a second status, which indicates that the cache line is owned by the processor.

14. The apparatus according to claim 12, wherein, The cache line is requested by a node controller located remotely from the standby node, and the data migration logic is used for: After writing the acquired cache line, the directory status of the cache line is updated to a third state, which indicates that the cache line is owned by the remote node controller.

15. The apparatus according to claim 10, wherein, The cache line is requested by the local processor of the standby node, which triggers an exception and causes exception handlers residing on different nodes in the NUMA system to notify the node controller of the standby node of the address of the corrupted cache line.

16. The apparatus according to claim 10, wherein, The cache line is requested by a node controller located remotely from the standby node, and the node controller of the standby node forwards the request to a corresponding processor local to the standby node, the corresponding processor determining that the requested cache line is corrupt.

17. The apparatus according to claim 10, in, The node controller initialization logic is used to allocate unused memory space to the node controller of the standby node, so as to allow the node controller to track the migration status of each cache line in the local memory of the standby node and set the migration status of each cache line to "not migrated"; as well as The data migration logic is used for: In response to determining that the migration status of the requested cache line in the local memory is set to "not migrated", the cache line is consistently obtained from the faulty node; Write the acquired cache line to the local memory; and Set the migration status of the written cache line to "migrated".

18. The apparatus of claim 10, further comprising a block transfer engine for progressively traversing the local memory to migrate unmigrated and unrequested cache lines.

Images